diff --git a/docs/docs/proposals/index.md b/docs/docs/proposals/index.md index eeb3ed9..7811c75 100755 --- a/docs/docs/proposals/index.md +++ b/docs/docs/proposals/index.md @@ -72,7 +72,9 @@ reference. One proposal in three stages, read in order: fuzz targets and benchmark gates are built for Rust. - [Stage 2 — core interface](shared-runtime-core/core-interface.md): the handshake, framings and connection state machine behind a stable C ABI and - WASM, built from one crate. *under discussion.* + WASM, built from one crate. ***accepted***, steps 1–3 built: the core + changes, the C ABI at 0.1 ([reference](../reference/c-abi.md)), and its + vectors, fuzzing and benchmarks. Bindings and ABI 1.0 are next. - [Stage 3 — generated codecs](shared-runtime-core/generated-codecs.md): native per-type codecs and protocol glue rendered from one plan in the codegen kernel. *under discussion.* diff --git a/docs/docs/proposals/shared-runtime-core/core-interface.md b/docs/docs/proposals/shared-runtime-core/core-interface.md index 2eb8147..c1e8e88 100644 --- a/docs/docs/proposals/shared-runtime-core/core-interface.md +++ b/docs/docs/proposals/shared-runtime-core/core-interface.md @@ -1,6 +1,8 @@ # One protocol core: a C ABI and WASM from one crate -Status: **under discussion** · part of [Shared runtime core](index.md) · stage 2 +Status: **accepted** · part of [Shared runtime core](index.md) · stage 2 · +rollout steps 1–3 done (see [What's built](#whats-built)); steps 4–6 are future +work **The question:** what exactly the shared core owns, and what its boundary looks like, so that ten or more languages can bind it with no loss of @@ -17,8 +19,178 @@ So every choice below is judged on all three at once. Where a host makes a cost unavoidable, the page says so, and the [stage 1 gates](verification.md) decide by measurement. +## What's built + +Built 2026-10-25 and 2026-10-26, in [`runtime`](https://github.com/ComlineProject/runtime) except where noted. These +are [rollout](#rollout) steps 1–3. The as-built API is the +[C ABI reference](../../reference/c-abi.md). + +**Step 1: the core changes** + +- **JSON-RPC without `std`** ([runtime#49](https://github.com/ComlineProject/runtime/pull/49)): a `jsonrpc` + feature that needs `alloc` and no `std`. Request ids are written by a stack + writer, not through `to_string`. +- **Body-bytes entry points** ([runtime#50](https://github.com/ComlineProject/runtime/pull/50)): + `encode_request_body` on `Framing` and `ConnectionFraming`. `Call` is + unchanged. +- **The stream layer and the handshake step** + ([runtime#51](https://github.com/ComlineProject/runtime/pull/51)): + - `contract::stream`: `DEFAULT_MAX_FRAME` (16 MiB), `put_prefix` and + `split_frame`; + - `CountBuf`; + - `HandshakeFrame::from_facts` and `check_bytes`; + - `RuntimeError::FrameTooLarge`. +- **Fixed room for waiting calls** ([runtime#52](https://github.com/ComlineProject/runtime/pull/52)): + - `State::with_capacity` and `unordered_with_capacity`; + - `RuntimeError::Busy` and `BufferTooSmall`; + - `State::encode_request_body`, which commits nothing on overflow. +- **Frame limits in the transports** ([runtime#55](https://github.com/ComlineProject/runtime/pull/55)): the + transport owns the limit, with one default for everyone. See + [the frame-limit design](#what-changed-from-the-proposal). +- **The simulator** ([simulator#29](https://github.com/ComlineProject/simulator/pull/29)) + frames through the runtime's JSON-RPC and `encode_request_body`. Its vendored + JSON-RPC copy and `Preencoded` are deleted. It still keeps its own request + ids instead of using `State`, because it plays both ends on one simulated + clock. + +**Step 2: the C ABI** ([runtime#53](https://github.com/ComlineProject/runtime/pull/53)), crate +`comline-runtime-ffi`: + +- ABI 0.1, with `include/comline.h` generated by cbindgen 0.29.2 and checked + in. CI fails when it drifts. +- A `cdylib` and a `staticlib` natively, and the same exports on wasm32, plus + `comline_wasm_reserve_in` and `comline_wasm_reserve_out`. +- Status codes, an opaque `comline_connection`, a 48-byte `comline_event`, the + `NAME_ESCAPED` flag and `comline_method_name_unescape`. +- Every function runs under `catch_unwind` and poisons the handle on a panic. + Nothing allocates after `comline_connection_new`, except `jsonrpc@1` + decoding (see [Found along the way](#found-along-the-way)). A call with + overlapping buffers, or a length over `isize::MAX`, is refused. +- CI jobs `ffi` and `ffi miri`. + +**Step 3: proving the boundary** + +- **The vectors, from Rust, C and Node** ([runtime#56](https://github.com/ComlineProject/runtime/pull/56)): + the `export_vectors` example, a C harness (static, dynamic, and under ASan and + UBSan), the JS glue `js/comline.mjs` with `comline.d.mts` (no `wasm-bindgen`), + and a Node harness. The same 65 cases pass in all three. +- **Fuzzing and a differential test** ([runtime#57](https://github.com/ComlineProject/runtime/pull/57)): the + `ffi_feed` and `ffi_vs_core` fuzz targets, and `tests/differential.rs`. +- **Benchmarks** ([runtime#59](https://github.com/ComlineProject/runtime/pull/59)): the C ABI's calls against the + same work on the core, gated with the runtime's. +- **Cheaper crossing** ([runtime#60](https://github.com/ComlineProject/runtime/pull/60)): every ABI call costs + fewer instructions. + +[Verification](verification.md#whats-built) has the detail of step 3's checks. +The `main` ruleset requires `ffi`, `ffi miri`, `wasm32 core` and the fuzz +targets; see [Branch rules](../../conduct/branch-rules.md). + +## What changed from the proposal + +The implementation changed these, each checked against the source. Everything +else on this page is as proposed. + +- **Method names follow [runtime#45](https://github.com/ComlineProject/runtime/pull/45).** The sketch said `name` must + be an identifier, and that a received method with an escape fails to decode. + As built: + - `name` is any UTF-8 text, and `jsonrpc@1` writes it escaped as serde_json + does. A name that isn't UTF-8 is `COMLINE_ERR_INVALID_ARGUMENT`; + - a received name is kept as the frame spelled it, and an event sets + `COMLINE_EVENT_NAME_ESCAPED` when it has escapes; + - `comline_method_name_unescape` turns it into UTF-8. +- **`Call` is unchanged.** The proposal borrowed `Call::name` (`&'a str`). The + new `encode_request_body(call_id, name: &str, …)` takes the name as a + borrowed `&str` instead, so `Call` keeps `&'static str` and nothing cascades + into generated code. +- **`COMLINE_ERR_BUSY` (23) is `max_waiting`'s overflow.** The proposal left it + open whether it is an error or the binding's job to queue. It is an error. + The request spends no id and writes nothing, and queueing is the binding's + choice. A one-way call needs no room, and `max_waiting` 0 serves requests and + sends one-way calls only. The code wasn't in the sketch. +- **`BUFFER_TOO_SMALL` reports the exact size with the connection unchanged.** + `CountBuf` measures the frame, and `State::encode_request_body` commits no id + and no waiting call on overflow. `*written` counts a stream's prefix too. + `start`, `encode_reply`, `handshake_encode` and `comline_method_name_unescape` + report their sizes the same way. +- **The frame-limit design.** It follows + [runtime#55](https://github.com/ComlineProject/runtime/pull/55): + - the transport owns the limit, with one default, 16 MiB, declared once in + `contract::stream`; + - in the ABI the limit is `comline_config.max_frame`: 0 is the default, and it + may be raised to `u32::MAX` less the prefix; + - sending over the limit is recoverable: nothing is written, no id is spent, + the connection works on; + - receiving over it closes the transport, in stream mode. In message mode the + frame is refused and the connection works on; + - the Rust drivers answer an oversized handler reply with a `Fault`. The ABI + reports `COMLINE_ERR_FRAME_TOO_LARGE` from `encode_reply` and leaves that + choice to the host; + - there is no `config.idp` key for it. +- **One `encode_reply`, built on the envelope.** The lockstep `Framing` has + three (`encode_response_ok`, `_err` and `_fault`). The ABI has one, as + `ConnectionFraming::encode_reply` does, which takes the envelope kind. A + `Fault` takes no body. +- **One-way depends on the mode, and a connection uses only the connection + framings.** + - `datagram@1` is lockstep and has no connection form, so + `comline_connection_new` returns `COMLINE_ERR_UNKNOWN_FRAMING` for it. The + ABI carries `datagram@2` and `jsonrpc@1`. + - `datagram@2` has no way to say a call is one-way. On `jsonrpc@1` a one-way + call is a notification, with no `id`. + - An event sets `COMLINE_EVENT_ONE_WAY` only when the frame says so, never from + a request id of 0. +- **Which fields each framing reads** is stated per function and set out in + [the reference](../../reference/c-abi.md#what-each-framing-reads). + `datagram@2` reads `protocol` and `call_id`. `jsonrpc@1` reads `name`, writes + an error's `error_id` as the JSON-RPC `code`, and writes a fault as `-32603`. +- **JSON-RPC without `std` is confirmed.** `serde_json` with `raw_value` builds + without `std`, and CI's `wasm32 core` job builds `alloc,jsonrpc` for + `wasm32-unknown-unknown`. +- **Smaller differences from the sketch,** all in `comline.h`: + - `comline_status` is an `int32_t` with constants, not an enum; + - names are `const uint8_t *` with a length, not `const char *`; + - `max_waiting` is a `uint32_t` and `max_frame` a `uint64_t`; + - `comline_config` has a `_reserved` byte that must be 0; + - `comline_event` carries `capabilities`, for the handshake event; + - a `comline_handshake` struct is what `comline_handshake_check` returns; + - the constants are `COMLINE_DEFAULT_MAX_FRAME` and + `COMLINE_MAX_FRAME_CEILING`, where the sketch had `COMLINE_MAX_FRAME`. +- **The JS glue does not catch a panic.** The sketch said it would. The trap is + thrown as a `WebAssembly.RuntimeError` and the host discards the instance. + The glue also reserves its regions on every call, not up front, and returns + copies, so no view of wasm memory outlives a call. +- **Deferred:** + - `comline_connection_size` and `comline_connection_init` come after 1.0, as a + minor addition; + - `comline@2`'s negotiation comes before 1.0 if it changes the event struct. + +### Found along the way + +- **[runtime#58](https://github.com/ComlineProject/runtime/issues/58): `jsonrpc@1` decoding allocates after + `comline_connection_new`.** The ABI promises no allocation, but the framing + parses with serde_json, which allocates: + - on any refused frame, because it boxes its errors; + - when a member name has an escape; + - when a value nests brackets three deep, as in `"params":[[1]]`, which is an + ordinary frame. + + The planned fix is a hand-written strict JSON scanner in the core that + allocates nothing, checked against serde_json by a differential fuzz target. + **It is deferred:** Comline's own formats come first, and JSON is left for + later. The exemption is narrow and written down: the test + `jsonrpc_decode_allocates_nothing` is ignored, and the fuzz targets and the + differential proptest let through only those cases, on `jsonrpc@1` feeds. + Every `datagram@2` feed and every encode must still allocate nothing. + +The rest of this page is the proposal as written. Where the code has since +changed it, the text says so. + ## Where things stand +As of the proposal, before the work in [What's built](#whats-built). The simulator +and the core have both moved since: simulator#29 deleted the vendored JSON-RPC +and `Preencoded` named below. + The core already exists. It is `runtime/core` (crate `comline-runtime`) in `ComlineProject/runtime`, and most of what a shared core needs is already sans-IO, `no_std` and allocation-free. @@ -98,8 +270,9 @@ that is a host value.** - A sans-IO form of the exchange: "write our frame", then "check the peer's frame". The binding moves the bytes. 2. **Envelopes:** the tag byte, the `Err` ordinal, and `Fault`. -3. **All framings:** `datagram@1`, `datagram@2`, `jsonrpc@1`, and later - versions as they are written. +3. **All connection framings:** `datagram@2`, `jsonrpc@1`, and later versions + as they are written. (`datagram@1` is lockstep and has no connection form; + see [What changed](#what-changed-from-the-proposal).) - Request and reply encode go **around a body encoded elsewhere** (the `Preencoded` shape, made a first-class entry point). - Decode returns the body as a span of the input. @@ -111,7 +284,8 @@ that is a host value.** - `abandon` on timeout. The host's clock decides *when*; the core decides what a late reply means. 5. **Byte-level limits.** - - The stream length prefix (`u32` LE) and `MAX_FRAME`, declared once. + - The stream length prefix (`u32` LE) and the default frame limit, + declared once, in `contract::stream`. - The varint rules: shortest form only, and fits in `u16`. - JSON-RPC id range. Today these are per transport and per language. In the core they are one @@ -141,12 +315,16 @@ that is a host value.** ``` runtime/ ComlineProject/runtime ├── runtime/core/ comline-runtime: the reference, unchanged in role -└── runtime/ffi/ comline-runtime-ffi (new) +└── runtime/ffi/ comline-runtime-ffi ├── Cargo.toml crate-type = ["cdylib", "staticlib", "rlib"] ├── cbindgen.toml ├── include/comline.h generated by cbindgen, checked in ├── src/lib.rs the extern "C" surface: handles, spans, codes - └── js/ ES module glue for the wasm32 build, plus .d.ts + ├── js/ ES module glue for the wasm32 build, plus .d.mts + │ └── test/ the Node harness + ├── tests/ Rust tests, the C harness (tests/c), the differential test + ├── examples/ export_vectors: the vectors, written out for C and Node + └── bench/ the instruction-count benches (a package of its own) ``` - **One crate, two outputs.** On native targets it builds a `cdylib` (for @@ -160,136 +338,34 @@ runtime/ ComlineProject/runtime calls the same functions with the same semantics. The simulator uses `wasm-bindgen` because its surface is JS objects; this one isn't. - **It depends on `comline-runtime` without `std`** (`default-features = - false, features = ["alloc"]`, plus a `jsonrpc` feature once that moves; see + false, features = ["alloc", "jsonrpc"]`; see [What changes in the core](#what-changes-in-the-core)). So `package_abi` never enters the build, and wasm32 works before stage 5. The FFI crate itself uses `std` for `catch_unwind`, and wasm32 has `std`. - **The header is generated, then checked in.** - - cbindgen is pinned, and CI fails when `include/comline.h` differs from a - fresh run, as the docs' `cli` and `idl` drift jobs do. + - cbindgen is pinned (0.29.2), and CI's `ffi` job fails when + `include/comline.h` differs from a fresh run, as the docs' `cli` and `idl` + drift jobs do. - Reviewers then see every ABI change as a header diff. - **Why in `runtime` and not its own repo:** the FFI layer is a thin translation of `comline-runtime`. Splitting it would put every core change through a two-repo cascade. The core's tests, vectors and fuzz targets already live there. -### API sketch - -```c -/* comline.h (sketch). All integers are fixed width; no struct is - passed by value across the boundary except the small ones shown. */ - -#define COMLINE_ABI_MAJOR 0 /* 1 once frozen; see Rollout, step 5 */ -#define COMLINE_ABI_MINOR 1 - -uint32_t comline_abi_version(void); /* (major << 16) | minor */ - -typedef enum comline_status { /* append-only, stable values */ - COMLINE_OK = 0, - COMLINE_NEED_MORE = 1, /* stream mode: no whole frame yet */ - COMLINE_BUFFER_TOO_SMALL = 2, /* *written holds the size needed */ - COMLINE_ERR_FRAMING = 16, /* RuntimeError::Framing */ - COMLINE_ERR_HANDSHAKE = 17, /* RuntimeError::Handshake */ - COMLINE_ERR_FRAME_TOO_LARGE = 18, - COMLINE_ERR_UNKNOWN_FRAMING = 19, - COMLINE_ERR_INVALID_ARGUMENT = 20, /* null pointer, bad name bytes */ - COMLINE_ERR_STATE = 21, /* e.g. a call before the handshake */ - COMLINE_ERR_POISONED = 22, /* an earlier call panicked; drop the handle */ -} comline_status; - -typedef struct comline_connection comline_connection; /* opaque */ - -typedef struct comline_span { uint32_t offset; uint32_t len; } comline_span; - /* into the caller's input buffer */ - -typedef enum comline_event_kind { - COMLINE_EVENT_HANDSHAKE_OK = 0, - COMLINE_EVENT_REQUEST = 1, - COMLINE_EVENT_REPLY = 2, - COMLINE_EVENT_STALE = 3, -} comline_event_kind; - -typedef enum comline_envelope_kind { - COMLINE_ENVELOPE_OK = 0, COMLINE_ENVELOPE_ERR = 1, COMLINE_ENVELOPE_FAULT = 2, -} comline_envelope_kind; - -typedef struct comline_event { - uint8_t kind; /* comline_event_kind */ - uint8_t flags; /* HAS_PROTOCOL, ONE_WAY, CALL_BY_NAME, BODY_ABSENT */ - uint16_t protocol; /* valid with HAS_PROTOCOL */ - uint16_t call_id; /* valid without CALL_BY_NAME */ - uint8_t envelope; /* comline_envelope_kind, for REPLY */ - uint8_t _reserved; - uint16_t error_id; /* the schema-global ordinal, for ENVELOPE_ERR */ - uint64_t request_id; - comline_span call_name; /* valid with CALL_BY_NAME (jsonrpc@1) */ - comline_span body; /* params for REQUEST, payload or error body for REPLY */ - comline_span frame; /* the whole frame, for logging or forwarding */ -} comline_event; - -typedef struct comline_config { - uint64_t ir_hash; - const char* wire_format; size_t wire_format_len; /* "packed@1" */ - const char* framing; size_t framing_len; /* "datagram@2" */ - uint32_t capabilities; - uint8_t handshake; /* COMLINE_HANDSHAKE_COMLINE or _NONE */ - uint8_t stream; /* 1: u32 LE length-prefixed byte stream; 0: one frame per message */ - uint8_t unordered; /* 1: replies in any order (async hosts) */ - uint16_t max_waiting; /* waiting-call capacity, fixed at create */ - uint32_t max_frame; /* <= COMLINE_MAX_FRAME (16 MiB); 0 = the default */ -} comline_config; - -comline_status comline_connection_new(const comline_config* cfg, comline_connection** out); -void comline_connection_free(comline_connection* conn); -``` +### The API -The three representative calls: - -```c -/* 1. Feed bytes, get the next event. One frame per call. - * - * Stream mode: `in` is whatever the host has read so far. If a whole - * length-prefixed frame is there, it is decoded, *consumed is set to the - * prefix + frame length, and *ev describes it, with spans relative to `in`. - * Otherwise COMLINE_NEED_MORE, *consumed = 0, and ev->frame.len holds the - * total bytes needed once the prefix itself is complete (0 before that). - * - * Message mode: `in` is exactly one frame. - * - * The first frame on a COMLINE_HANDSHAKE_COMLINE connection is the peer's - * handshake: it is checked and yields HANDSHAKE_OK or COMLINE_ERR_HANDSHAKE. */ -comline_status comline_connection_feed( - comline_connection* conn, - const uint8_t* in, size_t in_len, - size_t* consumed, - comline_event* ev); - -/* 2. Frame a request around a body the generated codec already wrote. - * - * `name` is used by name-oriented framings (jsonrpc@1) and ignored by the - * datagram framings. It must be an identifier (no JSON escaping), or the - * call fails with COMLINE_ERR_INVALID_ARGUMENT. In stream mode the length - * prefix is written too, so `out` is ready for one transport write. - * - * On COMLINE_BUFFER_TOO_SMALL, *written is the size needed and the - * connection is unchanged: no request id is spent and nothing is waited on. */ -comline_status comline_connection_encode_request( - comline_connection* conn, - uint16_t protocol, uint16_t call_id, - const uint8_t* name, size_t name_len, - uint8_t expects_reply, - const uint8_t* body, size_t body_len, - uint8_t* out, size_t cap, size_t* written, - uint64_t* request_id); - -/* 3. Frame the reply to a request this side was given. */ -comline_status comline_connection_encode_reply( - comline_connection* conn, - uint16_t protocol, uint64_t request_id, - uint8_t envelope, uint16_t error_id, - const uint8_t* body, size_t body_len, - uint8_t* out, size_t cap, size_t* written); -``` +The header and the [C ABI reference](../../reference/c-abi.md) hold the +as-built API. The C sketch that stood here is gone. Its shape held: + +- a version, `comline_abi_version()`, and a header that carries it; +- an opaque `comline_connection`, made from a `comline_config`; +- `comline_connection_feed`, `comline_connection_encode_request` and + `comline_connection_encode_reply`, one whole frame per call; +- event spans as offsets into the caller's input; +- append-only status codes. + +[What changed](#what-changed-from-the-proposal) lists where the built API +differs from the sketch. The rest of the surface is small: @@ -297,10 +373,11 @@ The rest of the surface is small: handshake frame. - `comline_connection_abandon(conn, request_id)` gives up on a call after a timeout. -- `comline_connection_is_waiting(conn)`. +- `comline_connection_is_waiting(conn, waiting)`. - Free functions with no handle: `comline_name_hash`, `comline_handshake_encode` and `comline_handshake_check`, for hosts that - run the handshake themselves. + run the handshake themselves; `comline_supports`; and + `comline_method_name_unescape`. A call from Python, Go or Java is then: @@ -332,16 +409,18 @@ The FFI layer is for every other language, and Rust never pays for it. `comline_connection_new`, sized by `max_waiting`, and frees it in `comline_connection_free`. The host never frees library memory and the library never frees host memory. -- After the handle exists, the happy path allocates nothing. That needs - `State.waiting` to become a fixed-capacity array and `jsonrpc@1` to stop - formatting ids through `String`. Both are listed under - [What changes in the core](#what-changes-in-the-core), and both already - apply to Rust callers under - [§4.6's budget](../../design/core-target-contract.md#the-budget) - ("**zero** allocations on the happy path"). +- After the handle exists, the happy path allocates nothing. + - `State::with_capacity` reserves the waiting calls' room in + `comline_connection_new` (a `Vec` reserved to `max_waiting`, not an + array), and `jsonrpc@1` writes ids on the stack. Both apply to Rust + callers too, under + [§4.6's budget](../../design/core-target-contract.md#the-budget) + ("**zero** allocations on the happy path"). + - **One exemption, as built:** `jsonrpc@1` decoding allocates through + serde_json. See [Found along the way](#found-along-the-way). - A host with no allocator story of its own (embedded C, Zig) can get a later `comline_connection_size(cfg)` / `comline_connection_init(mem, …)` - pair. That is a minor ABI addition, not part of 1.0. + pair. That is a minor ABI addition, **after 1.0**, not part of it. **Panics never cross.** - Since Rust 1.81, a panic unwinding out of an `extern "C"` function aborts @@ -355,11 +434,13 @@ The FFI layer is for every other language, and Rust never pays for it. - Every later call on that handle returns the same code, so the binding closes the connection. - **On wasm32,** a panic is a trap (`WebAssembly.RuntimeError`). The JS glue - catches it and discards the instance. Unwinding on wasm32 needs the - exception-handling proposal, which this does not depend on. + does not catch it: the trap reaches the host, which discards the instance. + Unwinding on wasm32 needs the exception-handling proposal, which this does + not depend on. - **Panics are bugs, not a control path.** The decode paths are already - written as `Option`-returning `split_at_checked` chains. Stage 1's fuzz - targets run against the FFI entry points with "no panic" as the check. + written as `Option`-returning `split_at_checked` chains. The `ffi_feed` and + `ffi_vs_core` fuzz targets run against the FFI entry points and fail if a + panic crosses. **Thread safety.** - A `comline_connection*` is **movable, not shareable**. Any thread may use @@ -393,9 +474,10 @@ The FFI layer is for every other language, and Rust never pays for it. - **Status codes and flag bits are append-only.** A binding maps an unknown status to a generic error carrying the number. `RuntimeError` is already `#[non_exhaustive]`, for the same reason. -- `comline_supports(name, len)` answers whether this library build knows an - encoding name (`"datagram@2"`). A binding can then refuse a schema's - `@framing` at setup, not at the first frame. +- `comline_supports(name, len)` answers whether this library build carries a + framing or handshake name (`"datagram@2"`, `"jsonrpc@1"`, `"comline@1"`, + `"none"`). Wire formats are the host's codecs, so none is listed. A binding + can then refuse a schema's `@framing` at setup, not at the first frame. **Zero-copy where the host allows.** - **Input is borrowed.** Events point into it by `comline_span` offsets, so @@ -408,15 +490,20 @@ The FFI layer is for every other language, and Rust never pays for it. reallocates its buffer between the call and the decode. A garbage-collected host (Java heap arrays, .NET with pinning only during the call, Go) usually does exactly that. -- **`u32` spans are enough:** `MAX_FRAME` is 16 MiB. They are also the same - width on wasm32. +- **`u32` spans are enough:** the default frame limit is 16 MiB, and a + connection's limit can't pass `u32::MAX` less the prefix. They are also the + same width on wasm32. - **Two JSON-RPC edge cases surface as flags, not copies:** - - Absent `params` or `result` decode today as the literal `null` (a static + - Absent `params` or `result` decode as the literal `null` (a static slice, not part of the input). The event sets `BODY_ABSENT` with an empty - span, and the JSON codec reads `null`. + span, and the JSON codec reads `null`. A `null` the frame spelled out sets + it too, since the framing can't tell the two apart. - `serde_json` will not borrow a string that contains escapes, so a - `method` with a `\u` escape fails to decode today. The FFI keeps that - behaviour. A vector should pin it before anything changes it. + `method` with an escape failed to decode. **As built + ([runtime#45](https://github.com/ComlineProject/runtime/pull/45)):** the + name is kept as the frame spelled it and the event sets `NAME_ESCAPED`; + `comline_method_name_unescape` reads it. The `found-request-method-*` + vectors pin it. - **Output is written once,** straight into the caller's `out`. The body passed to `encode_*` is copied exactly once, into the frame. That is the same single copy the Rust path makes when `Packed` encodes into the send @@ -426,26 +513,28 @@ The FFI layer is for every other language, and Rust never pays for it. All of these are small. Each one also helps Rust callers. -| Change | Why | -|---|---| -| Body-bytes entry points on `Framing` / `ConnectionFraming` (`encode_request_body(…, body: &[u8], …)`), the generic `Serialize` form becoming a wrapper | The `Preencoded` shim, made first-class. The simulator can drop its copy. | -| `Call::name` borrowed (`&'a str`) rather than `&'static str` | Host-owned names; the simulator's `WireFraming` workaround goes away. | -| `jsonrpc@1` behind `alloc` or its own `jsonrpc` feature, not `std`; ids written without `to_string` | The simulator vendors it today. Whether `serde_json`'s `raw_value` builds without `std` has to be checked first. | -| `State.waiting` with a capacity fixed at construction | No allocation on the happy path; an overflow becomes an error code, not a reallocation. | -| A sans-IO stream deframer: the `u32` LE prefix plus one `MAX_FRAME` constant | Replaces the two declarations in `transport.rs` and the tokio `Tcp`. | -| A sans-IO handshake step (`ours.encode`, `check(peer_bytes)`) beside `ComlineHandshake` | `ComlineHandshake::exchange` is `Transport`-bound. | -| A counting `BufMut` | So `BUFFER_TOO_SMALL` can report the exact size needed without mutating state. | +| Change | Why | Status | +|---|---|---| +| Body-bytes entry points on `Framing` / `ConnectionFraming` (`encode_request_body(…, body: &[u8], …)`), the generic `Serialize` form becoming a wrapper | The `Preencoded` shim, made first-class. The simulator can drop its copy. | Done ([runtime#50](https://github.com/ComlineProject/runtime/pull/50)). The generic form is kept as it was; the simulator dropped `Preencoded` (simulator#29). | +| `Call::name` borrowed (`&'a str`) rather than `&'static str` | Host-owned names; the simulator's `WireFraming` workaround goes away. | **Not done, not needed.** `encode_request_body` takes `name: &str`, so `Call` is unchanged. | +| `jsonrpc@1` behind `alloc` or its own `jsonrpc` feature, not `std`; ids written without `to_string` | The simulator vendors it today. Whether `serde_json`'s `raw_value` builds without `std` has to be checked first. | Done ([runtime#49](https://github.com/ComlineProject/runtime/pull/49)): a `jsonrpc` feature. `raw_value` builds without `std`. | +| `State.waiting` with a capacity fixed at construction | No allocation on the happy path; an overflow becomes an error code, not a reallocation. | Done ([runtime#52](https://github.com/ComlineProject/runtime/pull/52)): `State::with_capacity`; overflow is `RuntimeError::Busy`. | +| A sans-IO stream deframer: the `u32` LE prefix plus one `MAX_FRAME` constant | Replaces the two declarations in `transport.rs` and the tokio `Tcp`. | Done ([runtime#51](https://github.com/ComlineProject/runtime/pull/51)): `contract::stream`. Both transports read it ([runtime#55](https://github.com/ComlineProject/runtime/pull/55)). | +| A sans-IO handshake step (`ours.encode`, `check(peer_bytes)`) beside `ComlineHandshake` | `ComlineHandshake::exchange` is `Transport`-bound. | Done (runtime#51): `HandshakeFrame::from_facts` and `check_bytes`. | +| A counting `BufMut` | So `BUFFER_TOO_SMALL` can report the exact size needed without mutating state. | Done (runtime#51, #52): `CountBuf`, and `BufMut::overflowed`. | ## WASM **The same API, compiled to `wasm32-unknown-unknown`.** - `comline-runtime-ffi` exports the `comline_*` functions as they are. - `js/` holds a small ES module, hand-written, with no generated glue. It - - instantiates the module; + - instantiates the module, and checks its ABI version and the + `comline_event` layout; - reserves an input region and an output region in linear memory, through two wasm-only exports (`comline_wasm_reserve_in(len)`, `comline_wasm_reserve_out(len)`) that return pointers into memory the - module owns; + module owns. It does so on every call, which costs nothing when the + region is already big enough; - reads `comline_event` with a `DataView`. - **A size reference point:** the simulator's 2026-09-03 spike, built on `comline-runtime` with `alloc` only, was an 88 KB `.wasm`. @@ -457,12 +546,12 @@ All of these are small. Each one also helps Rust callers. | Bytes from the socket (a `Uint8Array` on the JS heap) into linear memory | one `set()`, a memcpy linear in frame size, per frame in | | The call | a JS→WASM call; small in current engines, not free | | Spans back | read through `DataView`; no copy | -| The body, for the generated JS codec | read in place through a view on `memory.buffer`, **or** copied out if the codec keeps it past the next call | +| The body, for the generated JS codec | **As built:** a subarray of the `Uint8Array` the host fed, at the span's offset. The glue hands out no view of wasm memory. | | A frame out | the codec's body is copied in, the frame is copied out: two memcpys | -- **Views on `memory.buffer` are invalidated when memory grows.** The glue - reserves its regions up front and only grows between calls. Even so, a - generated codec must not hold a view across calls. +- **Views on `memory.buffer` are invalidated when memory grows.** As built, + the glue takes fresh views after every reserve and returns copies, so a + generated codec holds none across calls. - **For small frames this can lose** to hand-written TypeScript framing, which works on the socket's `Uint8Array` with no copy. A `datagram@2` request header is 12 bytes for a protocol number below 128; the copy and @@ -538,8 +627,9 @@ addition, not a replacement. ## Distribution -**This is a sketch and an open question.** It needs its own pass before -stage 2 ships. +**This is a sketch and an open question.** It needs its own pass before the +release workflow and prebuilt binaries ([rollout](#rollout) step 5). Nothing +in it is built. **Targets.** Prebuilt binaries per triple, from `runtime`'s release workflow: @@ -566,9 +656,9 @@ workflow: | Dart, PHP, Lua | native libs inside the package; specifics per ecosystem | **Reproducibility.** -- Release builds use a **pinned stable toolchain**. The runtime repo's - `rust-toolchain` currently says `nightly`, unpinned, and that can't stay - for shipped binaries. +- Release builds use a **pinned stable toolchain**. The runtime repo pins stable + 1.98.0 in `rust-toolchain.toml` ([runtime#30](https://github.com/ComlineProject/runtime/pull/30)), + which replaced an unpinned `nightly`. - Builds run with `--locked`, `--remap-path-prefix` and a fixed `SOURCE_DATE_EPOCH`, and the release publishes checksums. - Each binding records the core release it bundles in a file, as @@ -621,35 +711,43 @@ stays a separate idea. ## Open questions -- **Handle memory for no-allocator hosts:** whether - `comline_connection_size` / `comline_connection_init` belongs in 1.0 or - 1.1. -- **`max_waiting` overflow:** whether it is an error to the caller of - `encode_request`, as proposed, or the binding's job to queue. +**Settled by the build:** + +- **Handle memory for no-allocator hosts:** `comline_connection_size` / + `comline_connection_init` come after 1.0, as a minor addition. +- **`max_waiting` overflow:** an error to the caller of `encode_request`, + `COMLINE_ERR_BUSY`. Queueing is the binding's job. +- **Whether `jsonrpc@1`'s `serde_json` dependency fits the no-`std` FFI + build:** it does, as is. Its decoding allocates, which breaks the ABI's + no-allocation rule; that is [runtime#58](https://github.com/ComlineProject/runtime/issues/58), + deferred. + +**Still open:** + +- **Whether `comline@2`'s negotiation** (protocol numbers handed out at the + handshake) changes the event struct. If so, it should land before ABI 1.0 + freezes. - **`pyo3` vs `cffi` for Python,** and whether a second binding surface is ever worth it. Decided by gates, but the default needs picking. - **Go without cgo:** wazero against pure Go, measured on the same benchmarks. -- **Whether `jsonrpc@1`'s `serde_json` dependency** fits the no-`std` FFI - build as is, or needs a narrower parser for the envelope fields. - **Distribution:** everything flagged in [Distribution](#distribution). -- **Whether `comline@2`'s negotiation** (protocol numbers handed out at the - handshake) changes the event struct. If so, it should land before ABI 1.0 - freezes. ## Rollout -1. **The core changes** in `comline-runtime`, each with vectors: body-bytes - framing entry points, borrowed `Call::name`, `jsonrpc@1` out of `std`, - fixed-capacity `waiting`, the stream deframer with one `MAX_FRAME`, and - the sans-IO handshake step. The simulator moves onto them and deletes its - vendored JSON-RPC and `Preencoded`. -2. **`comline-runtime-ffi`** with `comline.h` at ABI `0.x`: native and wasm32 - builds, the header drift check, and FFI fuzz targets in stage 1's +Steps 1–3 are done ([What's built](#whats-built)). + +1. **The core changes** in `comline-runtime`, each with vectors: **done.** + Body-bytes framing entry points, `jsonrpc@1` out of `std`, fixed-capacity + `waiting`, the stream deframer with one default limit, and the sans-IO + handshake step. Borrowed `Call::name` was not needed. The simulator moved + onto them and deleted its vendored JSON-RPC and `Preencoded`. +2. **`comline-runtime-ffi`** with `comline.h` at ABI `0.x`: **done.** Native and + wasm32 builds, the header drift check, and FFI fuzz targets in stage 1's harness. 3. **A C test binding** in `runtime` CI runs the conformance vectors through - the C ABI and the wasm32 build. This proves the boundary before any - language depends on it. + the C ABI and the wasm32 build: **done.** C and Node replay the same 65 + cases as Rust. This proves the boundary before any language depends on it. 4. **The first two real bindings, chosen to stress the boundary:** - **TypeScript**, through WASM and napi: stage 4's question; - **Go**, through cgo: the most expensive call. @@ -668,13 +766,13 @@ stays a separate idea. - FFI fuzz targets; - the release workflow; - the workspace `Cargo.toml`, where the `runtime-langs/c` stub (an `add` - function today) is superseded. -- **Simulator:** drops its vendored JSON-RPC and the `Preencoded` shim once - the core has body-bytes entry points. + function) was removed by runtime#53. +- **Simulator:** dropped its vendored JSON-RPC and the `Preencoded` shim + (simulator#29). - **comline-typescript:** stage 4, behind the gates. - **Each future `comline-`:** a binding, its packaging, and the gates. - **Docs:** - the [runtime guide](../../guide/runtime/index.md)'s "no FFI boundary"; - [`core-target-contract.md`](../../design/core-target-contract.md) §4.7 (answered here) and §4.6's `no_std` layering table; - - a `comline.h` reference page. + - a `comline.h` reference page: done, the [C ABI reference](../../reference/c-abi.md). diff --git a/docs/docs/proposals/shared-runtime-core/index.md b/docs/docs/proposals/shared-runtime-core/index.md index 57ca4c8..18bd420 100644 --- a/docs/docs/proposals/shared-runtime-core/index.md +++ b/docs/docs/proposals/shared-runtime-core/index.md @@ -3,8 +3,9 @@ Status: **under discussion** · tracked in [runtime#29](https://github.com/ComlineProject/runtime/issues/29) · direction agreed 2026-10-25; stage 1 accepted and in progress (see -[what's built](verification.md#whats-built)); stages 2–3 are proposals below, -stages 4–5 are agreed +[what's built](verification.md#whats-built)); stage 2 accepted, with its first +three rollout steps built (see [what's built](core-interface.md#whats-built)); +stage 3 is a proposal below, stages 4–5 are agreed **The question:** today every Comline runtime is written from scratch in its own language, and every runtime feature is built once per language. That @@ -111,7 +112,7 @@ Split the work by what can cross a language boundary cheaply. | Stage | Page | State | |---|---|---| | 1 | [Verification: differential fuzzing and benchmark gates](verification.md) | accepted, in progress | -| 2 | [One protocol core: a C ABI and WASM from one crate](core-interface.md) | proposal | +| 2 | [One protocol core: a C ABI and WASM from one crate](core-interface.md) | accepted; rollout steps 1–3 built, bindings and ABI 1.0 next; see the [C ABI reference](../../reference/c-abi.md) | | 3 | [Codecs and protocol glue generated from one plan](generated-codecs.md) | proposal | | 4 | Move TypeScript onto the core where the benchmarks allow | agreed | | 5 | Clean up the leftovers and the documents that contradict this | agreed | diff --git a/docs/docs/proposals/shared-runtime-core/verification.md b/docs/docs/proposals/shared-runtime-core/verification.md index 5d08464..0be59bd 100644 --- a/docs/docs/proposals/shared-runtime-core/verification.md +++ b/docs/docs/proposals/shared-runtime-core/verification.md @@ -1,6 +1,7 @@ # Verification: differential fuzzing and benchmark gates -Status: **accepted** · part of [Shared runtime core](index.md) · stage 1 in progress +Status: **accepted** · part of [Shared runtime core](index.md) · stage 1 in progress; +the C ABI's checks (stage 2, steps 2–3) are built **The question:** with ten or more language runtimes and generators, how does each one stay byte-identical to the others, exactly as strict as the others @@ -121,6 +122,68 @@ except where noted. length; `json@1` and `jsonrpc@1` replaced invalid UTF-8 and skipped a byte-order mark; the JSON reader accepted an escaped lone surrogate. +**For the C ABI** (stage 2, steps 2–3 of the +[core interface](core-interface.md#whats-built)), 2026-10-26: + +- **Frame-limit vectors and a fuzz target** + ([runtime#55](https://github.com/ComlineProject/runtime/pull/55)): `vectors/stream.json`, 24 cases, and the + `stream_split` fuzz target, which checks `split_frame` and `put_prefix` + against each other and against the prefix's own rules. It also fixed the + transports' frame-limit bugs (see the + [core interface](core-interface.md#what-changed-from-the-proposal)). +- **The ABI's vectors, from Rust, C and Node** + ([runtime#56](https://github.com/ComlineProject/runtime/pull/56)): one harness per host, on the same cases and + with the same checks, against the six vector files the ABI can carry + (`name-hash`, `handshake`, `envelope`, `framing-datagram`, + `framing-datagram2`, `framing-jsonrpc`). + - The cases are 65. The nine `datagram@1` cases have no connection form, so + for them the harness checks that the ABI refuses it. + - C is built against `comline.h` and runs on the static library, the dynamic + library, and the static one under ASan and UBSan. Node runs against the + wasm32 build, through the JS glue. + - C and Node can't run a codec, so the `export_vectors` example writes the + cases out with their params already encoded. Each harness fails on a + mismatch, and on a case count that differs from the vectors'. + - CI job `ffi`; Miri runs the Rust side separately as `ffi miri`. +- **Two fuzz targets and a differential proptest** + ([runtime#57](https://github.com/ComlineProject/runtime/pull/57)): + - `ffi_feed` runs random configs and sequences (calls, replies, abandons, peer + frames, chunked deliveries, flipped bytes) through a `comline_connection` + and holds every call to the ABI's rules: no panic crosses, only documented + statuses, spans inside the frame, exact retry sizes, refusals spend no + request id, nothing allocates after `comline_connection_new`. + - `ffi_vs_core` adds the comparison: each call is also made through `State`, + the connection framings and `HandshakeFrame` directly, and must give the + same status, bytes, request id, `*consumed` and event. + - `tests/differential.rs` runs the same driver on proptest's sequences, with + 100,000 cases nightly. Both targets share the driver, so a sequence fails the + same way under either. + - The first nightly run with them (main at `b773057`) ran each target for 10 + minutes: 4,154,468 executions for `ffi_feed` and 2,465,895 for + `ffi_vs_core`, no findings, and the differential property passed. + - They found [runtime#58](#found-along-the-way), deferred. +- **FFI benchmarks and a gate over every target** + ([runtime#59](https://github.com/ComlineProject/runtime/pull/59), [runtime#60](https://github.com/ComlineProject/runtime/pull/60)): + - `runtime/ffi/bench` is a package of its own, so the ABI is built as a host + links it, without the crate's test hooks. Each call is measured twice: + through its `extern "C"` function (`ffi/abi/...`) and as the same work on + `State` and the framings (`ffi/core/...`). That is 104 benchmarks. + - The gate now runs every bench target, listed once as `TARGETS` in + `bench-report.py`. The limit (1%), the ledger and the rename rules are the + same for all. Ids are unique across targets: the runtime's keep the shared + definitions' names and every other target's starts with its name. + - A target the base doesn't have yet is recorded only, as is the whole report + when the base has no benches. + - Base and head now build at paths of the same length, because a longer path + moved 16 of the 102 FFI counts by a few instructions. + - The job summary pairs each ABI call with its core twin, the crossing cost. + It informs and isn't gated on its own. + - runtime#60 cut the crossing: every ABI call is cheaper, none dearer. The + numbers are in the [C ABI reference](../../reference/c-abi.md#what-crossing-costs). +- **Required checks.** The `main` ruleset requires `ffi`, `ffi miri`, + `wasm32 core` and the fuzz jobs `fuzz (ffi_feed)`, `fuzz (ffi_vs_core)` and + `fuzz (stream_split)`. See [Branch rules](../../conduct/branch-rules.md). + ## Found along the way Open issues the work turned up: @@ -140,6 +203,26 @@ Open issues the work turned up: name that needs one unescaped, producing invalid JSON. - [runtime#42](https://github.com/ComlineProject/runtime/issues/42): `jsonrpc@1` accepts invalid UTF-8 inside members it ignores. +- [runtime#58](https://github.com/ComlineProject/runtime/issues/58): `jsonrpc@1` decoding allocates after + `comline_connection_new`, which the C ABI promises it doesn't. The framing + parses with serde_json, which allocates: + - on any refused frame, because it boxes its errors; + - when a member name has an escape; + - when a value nests brackets three deep, as in `"params":[[1]]`, which is an + ordinary frame. + + **Deferred (decided 2026-10-26):** Comline's own formats come first, and JSON + is left for later. The planned fix is a hand-written strict JSON scanner in the + core that allocates nothing, with a differential fuzz target against + serde_json. Until then the exemption is narrow and written down: + - the test `jsonrpc_decode_allocates_nothing` (`runtime/ffi/tests/rules.rs`) + is ignored, and fails with `--ignored`; + - `serde_json_allocates` in `tests/common/drive.rs` lets the fuzz targets and + the differential proptest through on exactly those three cases, on + `jsonrpc@1` feeds only; + - every `datagram@2` feed and every encode must still allocate nothing; + - the fuzz README lists it under "Open", and the fix deletes the exemption and + un-ignores the test. - In TypeScript, each listed as a known divergence: [comline-typescript#73](https://github.com/ComlineProject/comline-typescript/issues/73) (`json@1` decoding doesn't check a value's shape against its type), @@ -161,6 +244,10 @@ Open issues the work turned up: `rust.bench.json` (every push to `main`) and `rust.fuzz.json` (nightly) to the runtime's `reports` branch, and the pages render them. TypeScript's benchmark results fill in when it checks in its `conformance-bench.json`. +- **The `jsonrpc@1` allocation** ([runtime#58](https://github.com/ComlineProject/runtime/issues/58)), deferred. +- **The wasm32 build and the JS glue are not benchmarked.** The ABI's benches + are native. A JavaScript call into WebAssembly, and the copies in and out, + are what [stage 4](index.md#stages) has to measure. - **A re-baseline path for the gate.** A correctness fix that costs time needs a deliberate way to move the baseline (see [Budgets](#budgets)). It isn't built yet. @@ -247,8 +334,9 @@ implementation. commit in `VECTORS_REV`, so a language's CI answers the same way on every run. Every language CI that vendors the corpus already has, or can install, a Rust toolchain (comline-typescript's CI has one for its codegen). -- **(b) comes later as an option, not a replacement.** Once stage 2 exists, a - language may link the reference in-process for speed. Codecs stay native +- **(b) comes later as an option, not a replacement.** Once stage 2 exists (the + C ABI is built, [runtime#53](https://github.com/ComlineProject/runtime/pull/53); no language's fuzzer links it + yet), a language may link the reference in-process for speed. Codecs stay native ([stage 3](generated-codecs.md)), so the differential check stays meaningful there. For framing it then checks the binding, not the logic. - **(c) falls out of (a)** for free: the reference can dump a batch of cases to a @@ -446,6 +534,10 @@ The last row is what [stage 4](index.md#stages) needs. For small frames, a TypeScript call into WASM or napi may cost more than the work it replaces. Only this row can say where that line is. +As built ([runtime#59](https://github.com/ComlineProject/runtime/pull/59)), the row covers the native ABI only: `ffi/abi/...` +against `ffi/core/...`, in both stream and message mode, and at the payload +sizes above for the calls that carry a frame. The wasm32 build and the JS glue aren't measured yet. + ### Judging a regression without flaky CI Shared CI runners are noisy. Wall-time differences of several percent between diff --git a/docs/docs/reference/c-abi.md b/docs/docs/reference/c-abi.md new file mode 100644 index 0000000..aefa533 --- /dev/null +++ b/docs/docs/reference/c-abi.md @@ -0,0 +1,437 @@ +# C ABI + +The C ABI of `comline-runtime-ffi`: what a language other than Rust binds to +for the `comline@1` handshake, the connection framings (`datagram@2`, +`jsonrpc@1`), envelopes, connection state and frame limits. Codecs, +transports, async and dispatch stay in each host. + +- **ABI version 0.1.** It is not frozen; see [Version](#version). +- **Source:** [`runtime/ffi`](https://github.com/ComlineProject/runtime/tree/main/runtime/ffi) + in the runtime repository. +- **Header:** [`include/comline.h`](https://github.com/ComlineProject/runtime/blob/main/runtime/ffi/include/comline.h). + cbindgen 0.29.2 generates it from `src/lib.rs`, and it is checked in. Its + comments are the doc comments of the Rust source. +- **Why it exists and what was decided:** [one protocol core](../proposals/shared-runtime-core/core-interface.md). + +This page was written against runtime `main` at `cb41d31`. Where it and the +header disagree, the header is right. + +## Version + +- The header defines `COMLINE_ABI_MAJOR` (0) and `COMLINE_ABI_MINOR` (1). +- `comline_abi_version()` returns the library's, as `(major << 16) | minor`. +- **A host checks it at load, before any other call.** + + ```c + if (comline_abi_version() != ((COMLINE_ABI_MAJOR << 16) | COMLINE_ABI_MINOR)) { + /* refuse the library, naming both versions */ + } + ``` + +- **Before 1.0, a minor counts as a major.** A host built against `0.1` accepts + only `0.1`. The JS glue does exactly this. +- The major changes when a signature, a struct layout or the meaning of a code + changes. The minor changes when functions, codes or flag bits are added. +- Structs are frozen within a major. New data comes through new functions or new + flag bits. +- Status codes, event kinds and flag bits are append-only. A host maps a code + it doesn't know to a generic error carrying the number. +- `comline_supports(name, len)` says whether this build carries a framing or + handshake by name, so a host can refuse a schema's `@framing` at setup, not + at the first frame. + +## A connection's life + +1. `comline_connection_new` with a [`comline_config`](#comline_config). +2. `comline_connection_start` writes this side's handshake frame. Send it + first. +3. `comline_connection_feed` with what arrives. The first frame is the peer's + handshake and gives `COMLINE_EVENT_HANDSHAKE_OK`. It may arrive before or + after `start`. A request or reply needs both. +4. Calls and replies: `comline_connection_encode_request`, + `comline_connection_feed`, `comline_connection_encode_reply`. When the + host's clock says a call timed out, `comline_connection_abandon`. +5. `comline_connection_free`. + +With `COMLINE_HANDSHAKE_NONE` there is nothing to send or check: the +connection is open from the start, and `start` writes nothing. + +## Status codes + +`comline_status` is an `int32_t`. Below 16 is an outcome, 16 and up is an error. + +| Code | Value | Meaning | +|---|---|---| +| `COMLINE_OK` | 0 | Done. | +| `COMLINE_NEED_MORE` | 1 | Stream mode: no whole frame in the input yet. Nothing was consumed. `ev->frame.len` is the total bytes the frame takes, prefix included, once the prefix is complete (0 before that). | +| `COMLINE_BUFFER_TOO_SMALL` | 2 | The output didn't fit `cap`. `*written` is the exact size needed, and nothing changed: no request id spent, nothing waits. | +| `COMLINE_ERR_FRAMING` | 16 | A malformed frame, or (ordered connections) a reply that skips the innermost waiting call. The frame was consumed and the connection stays usable. | +| `COMLINE_ERR_HANDSHAKE` | 17 | The peer's `comline@1` frame didn't parse, or names another schema, wire format or framing. The connection is closed. | +| `COMLINE_ERR_FRAME_TOO_LARGE` | 18 | A frame over `max_frame`. See [Frame limits](#frame-limits). | +| `COMLINE_ERR_UNKNOWN_FRAMING` | 19 | A framing name this library has no connection form of. | +| `COMLINE_ERR_INVALID_ARGUMENT` | 20 | A null pointer where one isn't allowed, a name that isn't UTF-8, a field outside its values, buffers that overlap, or a length over `isize::MAX`. | +| `COMLINE_ERR_STATE` | 21 | The call doesn't fit the connection's state: a request or reply before the handshake completed, a second `start`, or any call on a closed connection. | +| `COMLINE_ERR_POISONED` | 22 | An earlier call on this handle panicked. Drop the handle. | +| `COMLINE_ERR_BUSY` | 23 | The request wasn't made: `max_waiting` calls already wait. Nothing written, no request id spent. Queueing is the host's choice. | + +Every call that takes a handle can also return `COMLINE_ERR_INVALID_ARGUMENT` +(a null handle or pointer), `COMLINE_ERR_POISONED`, and `COMLINE_ERR_STATE` on a +closed connection. The tables below list only what is particular to a call. + +## Constants + +| Constant | Value | Meaning | +|---|---|---| +| `COMLINE_DEFAULT_MAX_FRAME` | 16 MiB | The frame limit when `max_frame` is 0, prefix not counted. | +| `COMLINE_MAX_FRAME_CEILING` | 4294967291 | The most `max_frame` can be: `u32::MAX` less the prefix. | +| `COMLINE_PREFIX_LEN` | 4 | A stream's length prefix: a `u32`, little-endian. | +| `COMLINE_HANDSHAKE_LEN` | 31 | The `comline@1` frame. | +| `COMLINE_HANDSHAKE_COMLINE` / `_NONE` | 0 / 1 | `comline_config.handshake`. | +| `COMLINE_ENVELOPE_OK` / `_ERR` / `_FAULT` | 0 / 1 / 2 | A reply's outcome. | + +## Structs + +### `comline_config` + +Read once by `comline_connection_new`. The name pointers are borrowed for that +call. Zero every field you don't set, including `_reserved`. + +| Field | Type | Meaning | +|---|---|---| +| `ir_hash` | `uint64_t` | The schema fingerprint the generated code was built from. Only the handshake reads it. | +| `wire_format`, `wire_format_len` | `const uint8_t *`, `size_t` | The wire format's name (`"packed@1"`), UTF-8, not NUL-terminated. The codecs are the host's, so any name is taken; only the handshake reads it, hashed. | +| `framing`, `framing_len` | `const uint8_t *`, `size_t` | `"datagram@2"` or `"jsonrpc@1"`. UTF-8, not NUL-terminated. | +| `max_frame` | `uint64_t` | The largest frame sent or taken, prefix not counted. 0 is the default. Above `COMLINE_MAX_FRAME_CEILING` is `COMLINE_ERR_INVALID_ARGUMENT`. | +| `capabilities` | `uint32_t` | Transport capability bits, sent in the handshake. Advisory: peers that differ still agree. | +| `max_waiting` | `uint32_t` | How many calls may wait for a reply at once. The room is reserved when the handle is made. 0 is none: the connection serves requests and sends one-way calls only. | +| `handshake` | `uint8_t` | `COMLINE_HANDSHAKE_COMLINE` or `COMLINE_HANDSHAKE_NONE`. | +| `stream` | `uint8_t` | 1: a byte stream, each frame behind a `u32` LE length prefix (TCP, a pipe, stdio). 0: one frame per message, boundaries kept by the transport (WebSocket, UDP). | +| `unordered` | `uint8_t` | 1: replies may come in any order (a host with several calls in flight). 0: calls nest, and a reply must answer the innermost waiting call. | +| `_reserved` | `uint8_t` | Must be 0. | + +`stream` and `unordered` take 0 or 1; any other value is +`COMLINE_ERR_INVALID_ARGUMENT`. The struct is 64 bytes on a 64-bit target and 48 +on wasm32, because it holds pointers. + +### `comline_span` + +`{ uint32_t offset; uint32_t len; }`: a part of the input, `offset` bytes from +its start. 8 bytes. + +### `comline_event` + +What one fed frame means. 48 bytes, aligned to 8. A field that isn't valid for +the kind and flags is 0. + +| Offset | Field | Type | Valid for | +|---|---|---|---| +| 0 | `kind` | `uint8_t` | always: `COMLINE_EVENT_*` | +| 1 | `flags` | `uint8_t` | always: see the flags below | +| 2 | `protocol` | `uint16_t` | a request with `HAS_PROTOCOL` | +| 4 | `call_id` | `uint16_t` | a request without `CALL_BY_NAME` | +| 6 | `envelope` | `uint8_t` | a reply: `COMLINE_ENVELOPE_*` | +| 7 | `_reserved` | `uint8_t` | | +| 8 | `error_id` | `uint16_t` | a reply with `COMLINE_ENVELOPE_ERR`: the schema-global ordinal | +| 10 | `_reserved2` | `uint16_t` | | +| 12 | `call_name` | `comline_span` | a request with `CALL_BY_NAME` | +| 20 | `body` | `comline_span` | a request's params, or a reply's result or error body | +| 28 | `frame` | `comline_span` | every frame: the whole frame, prefix not included | +| 36 | `capabilities` | `uint32_t` | `HANDSHAKE_OK`: the peer's capability bits | +| 40 | `request_id` | `uint64_t` | a request's id (0 for a one-way `jsonrpc@1` request), to answer it with; or the id of the call a reply answers | + +Event kinds: + +| Kind | Value | Meaning | +|---|---|---| +| `COMLINE_EVENT_HANDSHAKE_OK` | 0 | The peer's handshake arrived and agrees with ours. | +| `COMLINE_EVENT_REQUEST` | 1 | The peer calls this side. | +| `COMLINE_EVENT_REPLY` | 2 | The reply to a call this side waits on, which no longer waits. | +| `COMLINE_EVENT_STALE` | 3 | A reply to a call nobody waits on any more (it was abandoned). Drop it. | + +Flags: + +| Flag | Bit | Meaning | +|---|---|---| +| `COMLINE_EVENT_HAS_PROTOCOL` | `1 << 0` | `protocol` holds the protocol the request is for. Set on `datagram@2` requests. A `jsonrpc@1` request names only its method. | +| `COMLINE_EVENT_ONE_WAY` | `1 << 1` | The request wants no reply. Set whenever the frame says so (a `jsonrpc@1` notification), never inferred from a request id of 0. `datagram@2` can't say it, so there the schema tells one-way calls apart. | +| `COMLINE_EVENT_CALL_BY_NAME` | `1 << 2` | The request names its call by `call_name`, not `call_id`. | +| `COMLINE_EVENT_BODY_ABSENT` | `1 << 3` | The frame has no body: JSON-RPC `params`, `result` or `error.data` absent or `null`. `body` is empty, and a JSON codec reads it as `null`. | +| `COMLINE_EVENT_NAME_ESCAPED` | `1 << 4` | `call_name` is JSON string text with escapes in it, as the frame spelled it (`app.Chat\/count`). Read it with `comline_method_name_unescape`, or compare it as JSON text. | + +### `comline_handshake` + +A decoded `comline@1` frame, written by `comline_handshake_check`. 32 bytes. + +| Field | Type | Meaning | +|---|---|---| +| `ir_hash` | `uint64_t` | The peer's schema fingerprint. | +| `wire_format` | `uint64_t` | The wire format's name, hashed (`comline_name_hash`). | +| `framing` | `uint64_t` | The framing's name, hashed. | +| `capabilities` | `uint32_t` | The peer's capability bits. | +| `_reserved` | `uint32_t` | 0. | + +## Functions + +### Connection + +| Function | Purpose | Errors | +|---|---|---| +| `comline_connection_new(cfg, &out)` | Make a connection into `*out`. The one allocation the library makes: the handle, with room for `max_waiting` waiting calls. | `INVALID_ARGUMENT` (a null pointer, a name that isn't UTF-8, a field outside its values, `max_frame` above the ceiling, `*out` overlapping the config or its names), `UNKNOWN_FRAMING`. `*out` is null on any error, or left as it was if it overlaps an input. | +| `comline_connection_free(conn)` | Free it. Null is a no-op. Works on a poisoned or closed handle. | none | +| `comline_connection_start(conn, out, cap, &written)` | Write this side's handshake frame, with its prefix in stream mode. Without a handshake it writes nothing. | `BUFFER_TOO_SMALL` (`*written` is 31, or 35 in stream mode), `FRAME_TOO_LARGE` (`max_frame` below 31), `STATE` (already written). | +| `comline_connection_feed(conn, in, in_len, &consumed, &ev)` | Feed received bytes and get the event for the next frame. One frame per call. | `NEED_MORE`, `FRAMING`, `HANDSHAKE` (closes the connection), `FRAME_TOO_LARGE`. | +| `comline_connection_encode_request(conn, protocol, call_id, name, name_len, expects_reply, body, body_len, out, cap, &written, &request_id)` | Frame a request around a body the host's codec already wrote. `*request_id` is the id it was given. `expects_reply` is 1 for a call that waits, 0 for a one-way call. | `BUFFER_TOO_SMALL`, `FRAME_TOO_LARGE`, `BUSY`, `INVALID_ARGUMENT` (a name that isn't UTF-8, `expects_reply` not 0 or 1), `STATE` (handshake not complete). None of them spends a request id or leaves a call waiting. | +| `comline_connection_encode_reply(conn, protocol, request_id, envelope, error_id, body, body_len, out, cap, &written)` | Frame the reply to a request this side was given. | `BUFFER_TOO_SMALL`, `FRAME_TOO_LARGE`, `INVALID_ARGUMENT` (a `FAULT` with a body, an unknown envelope), `STATE` (handshake not complete). | +| `comline_connection_abandon(conn, request_id)` | Stop waiting on a call that timed out. A late reply to it is `STALE`. An id nobody waits on is a no-op. | none | +| `comline_connection_is_waiting(conn, &waiting)` | `*waiting` is 1 if any call waits for a reply, else 0. | none | + +### Without a handle + +These are pure and re-entrant. + +| Function | Purpose | Errors | +|---|---|---| +| `comline_abi_version()` | The ABI version: the major in the high 16 bits, the minor in the low 16. | none | +| `comline_supports(name, len)` | 1 if this build carries the named framing or handshake: `datagram@2`, `jsonrpc@1`, `comline@1`, `none`. 0 otherwise, `datagram@1` and wire formats included. | none | +| `comline_name_hash(name, len, &hash)` | The FNV-1a-64 hash of an encoding name, as the handshake carries it. | `INVALID_ARGUMENT` | +| `comline_handshake_encode(ir_hash, wire_format, wire_format_len, framing, framing_len, capabilities, out, cap, &written)` | Write a 31-byte `comline@1` frame with no prefix, for a host that runs the handshake itself. Any UTF-8 name is taken. | `BUFFER_TOO_SMALL` (`*written` is 31), `INVALID_ARGUMENT` | +| `comline_handshake_check(ours, ours_len, peer, peer_len, peer_out)` | Check a peer's frame against ours. They must agree on schema, wire format and framing; capability bits may differ. `*peer_out`, unless null, gets the peer's frame whenever it parses, so a host can say what the peer declared. Bytes after the 31-byte frame are ignored ([runtime#33](https://github.com/ComlineProject/runtime/issues/33)). | `HANDSHAKE` (they disagree, or `peer` doesn't parse), `INVALID_ARGUMENT` (`ours` doesn't parse) | +| `comline_method_name_unescape(in, len, out, cap, &written)` | Turn a `call_name` span with `NAME_ESCAPED` into UTF-8. Text with no escapes is copied as it is. | `BUFFER_TOO_SMALL` (`*written` is the exact size, nothing written), `INVALID_ARGUMENT` (not a valid JSON string's text) | + +## What each framing reads + +A connection always speaks a connection framing, whose frames say whether they +are a request or a reply. `datagram@1` is lockstep and has no connection form: +`comline_connection_new` returns `COMLINE_ERR_UNKNOWN_FRAMING` for it. + +| | `datagram@2` | `jsonrpc@1` | +|---|---|---| +| `encode_request` reads | `protocol` and `call_id`; `name` is ignored | `name`, escaped as serde_json escapes a string; `protocol` and `call_id` are ignored | +| A one-way request out | Looks like any other: the frame can't say it | A notification, with no `id` | +| An empty body is written as | nothing | `null` | +| `feed` of a request gives | `HAS_PROTOCOL`, `protocol`, `call_id`, `request_id` | `CALL_BY_NAME`, `call_name` (with `NAME_ESCAPED` if it has escapes), `request_id` | +| A one-way request in | not marked | A notification: `ONE_WAY`, `request_id` 0 | +| `encode_reply` writes | `protocol` and the request id | the id; `protocol` is ignored. An error is a JSON-RPC `error` object with `code` set to `error_id`; a fault is `-32603` | +| `BODY_ABSENT` | never | `params`, `result` or `error.data` absent or `null` | + +`name` is any UTF-8 text. It may be empty for `datagram@2`. + +## Frame limits + +`max_frame` is the connection's limit, in bytes, not counting a stream's +prefix. It may be lowered, or raised up to `COMLINE_MAX_FRAME_CEILING`. + +| Case | Result | +|---|---| +| An encode over the limit | `COMLINE_ERR_FRAME_TOO_LARGE`. Nothing written, no request id spent, the connection keeps working. | +| `feed`, stream mode, a prefix over the limit | `COMLINE_ERR_FRAME_TOO_LARGE`. The stream can't be resynchronised, so the connection is closed and later calls return `COMLINE_ERR_STATE`. | +| `feed`, message mode, a frame over the limit | `COMLINE_ERR_FRAME_TOO_LARGE`. The frame is refused and the connection keeps working. | + +`encode_reply` returns `FRAME_TOO_LARGE` for a reply over the limit and leaves +the connection usable. The Rust drivers answer such a reply with a fault; the +ABI leaves that to the host. + +## Rules every function keeps + +- **One crossing per frame.** `feed`, `encode_request` and `encode_reply` each + handle one whole frame. Nothing calls back into the host. +- **Buffers are the caller's.** Every output goes to an `out`, `cap`, + `*written` triple. Every input is borrowed for the call only. The library + never frees host memory and the host never frees library memory. +- **Spans are offsets, not pointers.** They count from the start of the input + passed to the `feed` that produced them, so they stay valid if the host + moves its buffer afterwards. They fit a `u32` under the frame ceiling. + Nothing is copied out of the input. +- **No allocation after `comline_connection_new`.** The handle is the one + thing the library allocates. One known exception, a bug: `jsonrpc@1` decoding, which + allocates through serde_json when it refuses a frame, when a member name has + an escape, and when a value nests brackets three deep (`"params":[[1]]`). It + is [runtime#58](https://github.com/ComlineProject/runtime/issues/58), deferred. + Every `datagram@2` feed and every encode, under either framing, allocates + nothing. +- **Null pointers.** A data pointer may be null only when its length or + capacity is 0. A length or capacity over `isize::MAX` is + `COMLINE_ERR_INVALID_ARGUMENT`. +- **Inputs and outputs must not overlap.** No buffer a call writes (`out`, + `*written`, `*ev`, the handle) may share a byte with any buffer or + out-pointer the same call takes. A call given overlapping ones returns + `COMLINE_ERR_INVALID_ARGUMENT` and touches nothing. Inputs may overlap each + other. +- **Panics never cross.** Every function runs inside `catch_unwind`. A panic is + a bug. The call returns `COMLINE_ERR_POISONED`, the handle is poisoned and + every later call on it returns the same, so the host drops the connection. + `comline_connection_free` still works. A free function returns + `COMLINE_ERR_POISONED` for the call that panicked. +- **Threads.** A handle may move between threads, but calls on one handle must + not overlap: the host locks it. Different handles are independent. +- **Refusals change nothing.** `BUFFER_TOO_SMALL`, `BUSY` and `FRAME_TOO_LARGE` + on an encode spend no request id and leave no call waiting. After + `BUFFER_TOO_SMALL`, retry with exactly `*written` bytes of room. +- **A closed connection** (a refused handshake, or an oversized stream prefix) + answers every call but `comline_connection_free` with `COMLINE_ERR_STATE`. + +The layouts of `comline_span`, `comline_event`, `comline_handshake` and +`comline_config` are asserted at compile time in `src/lib.rs`. + +## wasm32 and the JS glue + +The crate builds for `wasm32-unknown-unknown`, and the same `extern "C"` +functions are the module's exports. There is no `wasm-bindgen`: the `.wasm` is +the C ABI, so a non-JS WebAssembly host (wazero, wasmtime) calls the same +functions with the same meaning. + +```sh +cargo build -p comline-runtime-ffi --release --target wasm32-unknown-unknown +``` + +- **Layouts.** A pointer and `size_t` are 4 bytes, so `comline_config` is 48 + bytes. `comline_event` and `comline_handshake` don't change. +- **Two extra exports,** not in the header, on wasm32 without the `atomics` + feature: + - `comline_wasm_reserve_in(len)` and `comline_wasm_reserve_out(len)` return a + pointer to a region of at least `len` bytes that the module owns, or null if + memory can't grow. + - A host copies input into the `in` region and reads output from the `out` + region. + - Reserving more may move a region and grow memory, which detaches the host's + views. Take the pointer again after each reserve. +- **A panic is a trap.** The host discards the instance. + +[`js/comline.mjs`](https://github.com/ComlineProject/runtime/blob/main/runtime/ffi/js/comline.mjs) +and [`comline.d.mts`](https://github.com/ComlineProject/runtime/blob/main/runtime/ffi/js/comline.d.mts) +are a hand-written ES module over that build, with no dependencies. + +- **Thin.** Every method is one ABI call. It returns the status with whatever + the call wrote, retries nothing, never throws because of a status and keeps no + state of its own. A binding builds on it. +- **`instantiate(source)`** compiles the module, checks that its ABI version is + exactly the glue's, and feeds one `datagram@2` request through to check the + `comline_event` layout. +- **Memory.** Each call reserves its regions, copies inputs in and copies + outputs back, so a byte output is a fresh copy and nothing returned points + into wasm memory. A span is an offset into the array the host fed, so the + host reads a body from its own array. +- **Types.** A `u64` is a `BigInt` both ways. A name is a string, encoded as + UTF-8, or a `Uint8Array` of the bytes as they are. +- **A panic** is thrown as a `WebAssembly.RuntimeError`. The glue doesn't catch + it. + +```js +import { instantiate, Handshake } from "./comline.mjs"; + +const comline = await instantiate(await readFile("comline_runtime_ffi.wasm")); +const { status, connection } = comline.connectionNew({ + framing: "datagram@2", + handshake: Handshake.NONE, + maxWaiting: 16, +}); +``` + +## Building natively + +```sh +cargo build -p comline-runtime-ffi --release +``` + +This writes `libcomline_runtime_ffi.so` (a `cdylib`) and `libcomline_runtime_ffi.a` +(a `staticlib`) under `target/release/`. Linking the static library on Linux +needs what a Rust `staticlib` needs from the system: `-lgcc_s -lutil -lrt +-lpthread -lm -ldl -lc`. + +To regenerate the header, from `runtime/ffi`: + +```sh +cargo install cbindgen --version 0.29.2 --locked +cbindgen --config cbindgen.toml --output include/comline.h +``` + +## How it is verified + +- **The conformance vectors, from three hosts, on the same cases.** Rust + (`tests/vectors.rs`, through the `extern "C"` functions only), C + (`tests/c/vectors.c`, C11 against `comline.h`) and Node (`js/test/vectors.test.mjs`, + against the wasm32 build). C runs against the static library, the dynamic + library, and the static library under ASan and UBSan. + - The cases are every one in `name-hash` (8), `handshake` (9), `envelope` + (8), `framing-datagram` (9), `framing-datagram2` (12) and `framing-jsonrpc` + (19): 65 in all. Each harness fails on any mismatch, and on a case count + that differs from the vectors'. + - The nine `datagram@1` cases have no connection form. For them the harness + checks that the ABI refuses `datagram@1`. + - C and Node have no codec, so `examples/export_vectors.rs` writes the cases + out with their params already encoded. +- **The ABI against the core, on random sequences.** `tests/differential.rs` + runs random configs and sequences (calls, replies, abandons, peer frames, + chunked deliveries, flipped bytes) through a `comline_connection` and through + `State` and the framings directly. They must give the same frames, events and + refusals. Every call is also held to the rules above. The nightly run uses + 100,000 cases. +- **Fuzz.** `ffi_feed` checks the rules alone; `ffi_vs_core` also compares with + the core. Both share their driver with the proptest. +- **Miri.** The ABI's tests run under Miri, which checks aliasing, bounds and + leaks at the `extern "C"` boundary. +- **Header drift.** CI regenerates `comline.h` with the pinned cbindgen and + fails on any difference. +- **Benchmarks.** Instruction counts of each call, below. + +The CI jobs are `ffi`, `ffi miri`, `wasm32 core` and `fuzz (ffi_feed)`, +`fuzz (ffi_vs_core)` and `fuzz (stream_split)`; all are required to merge. See +[Verification](../proposals/shared-runtime-core/verification.md) for how the +fuzz targets and the benchmark gate work, and [Branch rules](../conduct/branch-rules.md). + +## What crossing costs + +The ABI's benches count instructions (Callgrind, through `iai-callgrind`) for +each call through its `extern "C"` function (`ffi/abi/...`) and for the same +work on `State` and the framings directly (`ffi/core/...`). The difference is +what crossing the boundary costs. + +These numbers are from the latest published `rust.bench.json` on the runtime's +[`reports`](https://github.com/ComlineProject/runtime/tree/reports) branch, which +has the `ffi/*` ids: main at `cb41d31`, the merge of +[runtime#60](https://github.com/ComlineProject/runtime/pull/60), built with +rustc 1.98.0. Message mode, a 16-byte payload: + +| Call | Core | ABI | Crossing | +|---|---:|---:|---:| +| `feed`, `datagram@2` | 141 | 294 | +153 | +| `feed`, `jsonrpc@1` | 1,550 | 1,703 | +153 | +| `encode_request`, `datagram@2` | 336 | 724 | +388 | +| `encode_request`, `jsonrpc@1` | 518 | 906 | +388 | +| `encode_reply`, `datagram@2` | 320 | 462 | +142 | +| `encode_reply`, `jsonrpc@1` | 331 | 473 | +142 | +| `name_hash` | 49 | 198 | +149 | +| `handshake_encode` | 437 | 772 | +335 | +| `handshake_check` | 100 | 138 | +38 | +| `handshake_check` with `peer_out` | 130 | 210 | +80 | + +- **The crossing is a constant per call, not per byte.** `feed` costs the same + extra at 16 B, 256 B, 4 KiB and 64 KiB. The two encode calls cost 48 more at + 64 KiB (436 and 190). +- **Stream mode** adds a few instructions to the core side and changes the + crossing little: `feed` +149, `encode_request` +388, `encode_reply` +141 at + 16 B. +- **`jsonrpc@1` decoding is its own cost.** It scans the frame with serde_json + at about 2.8 instructions a byte (185,825 for 64 KiB, on the core). The + crossing adds the same 153 on top. +- **runtime#60 cut it.** The overlap check was most of `encode_request`'s + crossing. From the PR's body, measured locally at the same paths: + `encode_request` `datagram@2` 965 to 742, `feed` `datagram@2` 319–327 to + 294–299, `handshake_check` 152 to 138. No call got dearer. Counts move by a + few instructions between machines, because the overlap check compares + addresses, so the CI figures above differ slightly from the PR's. +- **Not measured:** the wasm32 build and the JS glue. A JavaScript call into + WebAssembly and the copies in and out are the cost [stage 4](../proposals/shared-runtime-core/index.md#stages) + has to measure before moving TypeScript. + +The gate compares each call with its base on every pull request and fails at +a 1% rise, like the core's benches. Both sides of each pair are gated. + +## Not here yet + +- **`comline_connection_size` and `comline_connection_init`,** for a host that + supplies the handle's memory. After 1.0. +- **`comline@2`'s negotiation** hands protocol numbers out at the handshake and + may change the event struct. If it does, it lands before 1.0 freezes the ABI. +- **Bindings.** No language binds the ABI yet; the JS glue is the only host + code, and it is a thin wrapper, not a binding. diff --git a/docs/docs/reference/languages/index.md b/docs/docs/reference/languages/index.md index c1634f2..d6b7c27 100644 --- a/docs/docs/reference/languages/index.md +++ b/docs/docs/reference/languages/index.md @@ -108,6 +108,7 @@ commit. A runtime's suite runs every example against it: | envelope
`envelope` | ✅ supported | ✅ supported | | datagram framing
`framing-datagram` | ✅ supported | ✅ supported | | datagram@2 framing
`framing-datagram2` | ✅ supported | ✅ supported | +| stream framing
`stream` | ✅ supported | ❌ unsupported: no length-prefixed framing or frame limit yet: the transports are message-oriented ([comline-typescript#80](https://github.com/ComlineProject/comline-typescript/issues/80)) | | JSON-RPC framing
`framing-jsonrpc` | ✅ supported | ✅ supported | | JSON wire format
`format-json` | ✅ supported | 🟡 partial: decoding doesn't check enum variants or integer ranges ([comline-typescript#57](https://github.com/ComlineProject/comline-typescript/issues/57)) | | MessagePack wire format
`format-msgpack` | ✅ supported | ❌ unsupported: JSON and packed@1 ship | diff --git a/docs/docs/reference/languages/rust.md b/docs/docs/reference/languages/rust.md index 98aebf8..05cb080 100644 --- a/docs/docs/reference/languages/rust.md +++ b/docs/docs/reference/languages/rust.md @@ -63,6 +63,7 @@ Runtime `rust`. | envelope
`envelope` | a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault | ✅ supported | [8 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/envelope.json) | | datagram framing
`framing-datagram` | Comline's own binary framing of requests and responses | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram.json) | | datagram@2 framing
`framing-datagram2` | the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection | ✅ supported | [12 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram2.json) | +| stream framing
`stream` | frames on a byte stream (TCP, a pipe): each behind a u32 little-endian length prefix, refused over the limit, by default 16 MiB (16777216 bytes) in every runtime | ✅ supported | [24 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/stream.json) | | JSON-RPC framing
`framing-jsonrpc` | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [19 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) | | JSON wire format
`format-json` | values as UTF-8 JSON: a struct as an object | ✅ supported | [46 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-json.json) | | MessagePack wire format
`format-msgpack` | values as MessagePack: a struct as an array | ✅ supported | [45 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-msgpack.json) | @@ -78,9 +79,10 @@ are compared against (see [Verification](../../proposals/shared-runtime-core/ver ### Fuzzing and properties -- Four [`cargo-fuzz`](https://github.com/ComlineProject/runtime/tree/main/runtime/fuzz) - targets: `decode_no_panic`, `packed_canonical`, `framing_decode` and - `seed_vs_derive`. +- Seven [`cargo-fuzz`](https://github.com/ComlineProject/runtime/tree/main/runtime/fuzz) + targets: `decode_no_panic`, `packed_canonical`, `framing_decode`, + `seed_vs_derive` and `stream_split`, and, for the + [C ABI](../c-abi.md), `ffi_feed` and `ffi_vs_core`. - [Property tests](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/tests/properties.rs) (`proptest`) over random types and values, run with `cargo test`. - CI runs each target for a minute on every pull request, and for ten minutes diff --git a/docs/docs/reference/languages/typescript.md b/docs/docs/reference/languages/typescript.md index ee46c15..36fe672 100644 --- a/docs/docs/reference/languages/typescript.md +++ b/docs/docs/reference/languages/typescript.md @@ -63,6 +63,7 @@ Runtime `typescript`. | envelope
`envelope` | a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault | ✅ supported | [8 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/envelope.json) | | datagram framing
`framing-datagram` | Comline's own binary framing of requests and responses | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram.json) | | datagram@2 framing
`framing-datagram2` | the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection | ✅ supported | [12 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram2.json) | +| stream framing
`stream` | frames on a byte stream (TCP, a pipe): each behind a u32 little-endian length prefix, refused over the limit, by default 16 MiB (16777216 bytes) in every runtime | ❌ unsupported: no length-prefixed framing or frame limit yet: the transports are message-oriented ([comline-typescript#80](https://github.com/ComlineProject/comline-typescript/issues/80)) | — | | JSON-RPC framing
`framing-jsonrpc` | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [19 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) | | JSON wire format
`format-json` | values as UTF-8 JSON: a struct as an object | 🟡 partial: decoding doesn't check enum variants or integer ranges ([comline-typescript#57](https://github.com/ComlineProject/comline-typescript/issues/57))
fails: `format-json/decode-unknown-variant`, `format-json/decode-wider-integer-overflow` | [44 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-json.json) | | MessagePack wire format
`format-msgpack` | values as MessagePack: a struct as an array | ❌ unsupported: JSON and packed@1 ship | — | diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 3302169..ffcc99f 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -62,6 +62,9 @@ nav: - Wire formats: - packed@1: 'reference/wire/packed.md' + - Runtime: + - C ABI: 'reference/c-abi.md' + - Congregation: - Congregation: 'reference/congregation/index.md' - comline.toml: 'reference/congregation/comline-toml.md'