From c08a5005d07eb8b06ca96a751d32ec45f7af79fe Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 9 Nov 2026 19:30:58 +0800 Subject: [PATCH] Add the catalog of shared names catalog/ defines the guarantees, the facts and their values, the official encodings with their aliases and frozen spellings, and the wire features, with sets computed from them. specs/profile.cue takes each kind's facts and values from it, features.json is generated from it, and catalog/gen/catalog.json exports it for readers without CUE. check.sh also checks that profiles/ and vectors/ hold exactly what it names. The CUE module moves to the repository root so the specs can import it. No vector, profile or schema changes meaning. --- .github/workflows/check.yml | 2 +- README.md | 25 +- catalog/encodings.cue | 80 ++++++ catalog/facts.cue | 131 ++++++++++ catalog/features.cue | 55 ++++ catalog/gen/catalog.json | 458 ++++++++++++++++++++++++++++++++++ catalog/guarantees.cue | 29 +++ catalog/readme.md | 50 ++++ catalog/schema.cue | 96 +++++++ catalog/sets.cue | 35 +++ cue.mod/module.cue | 4 + features.json | 78 ++++-- specs/check.sh | 47 +++- specs/cue.mod/module.cue | 4 - specs/gen/profile.schema.json | 2 +- specs/profile.cue | 27 +- specs/readme.md | 13 +- 17 files changed, 1086 insertions(+), 50 deletions(-) create mode 100644 catalog/encodings.cue create mode 100644 catalog/facts.cue create mode 100644 catalog/features.cue create mode 100644 catalog/gen/catalog.json create mode 100644 catalog/guarantees.cue create mode 100644 catalog/readme.md create mode 100644 catalog/schema.cue create mode 100644 catalog/sets.cue create mode 100644 cue.mod/module.cue delete mode 100644 specs/cue.mod/module.cue diff --git a/.github/workflows/check.yml b/.github/workflows/check.yml index f4f72b2..c6a5782 100644 --- a/.github/workflows/check.yml +++ b/.github/workflows/check.yml @@ -11,7 +11,7 @@ jobs: name: specs runs-on: ubuntu-latest env: - # Keep in step with specs/cue.mod/module.cue. + # Keep in step with cue.mod/module.cue. CUE_VERSION: v0.17.1 steps: - uses: actions/checkout@v4 diff --git a/README.md b/README.md index 22915d3..4a3b536 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ What every Comline implementation is held to: the wire contract as byte-exact conformance vectors, the facts each official encoding states and how they're -proven, and the formats of these files. A runtime, a third-party encoding or a +proven, the formats of these files, and the catalog of names they share. A runtime, a third-party encoding or a tool implements against this repository, and pins it like any other dependency. Comline's own runtimes do the same. @@ -10,14 +10,21 @@ Version: **0.10** ([`VERSION`](VERSION)). While Comline is in its early days (0.x), a minor version may change anything, and the change is named in the pull request that makes it and in [the changes](#changes) below. -The shared vocabulary of names (encodings, guarantees, facts, features and -the sets they form) will join this repository too: see -[the proposal](https://github.com/ComlineProject/docs/pull/162) and -[core#219](https://github.com/ComlineProject/core/issues/219). +## The catalog + +[`catalog/`](catalog/readme.md) defines once every name that more than one +program has to agree on: the guarantees, the facts and the values each may +take, the official encodings with their aliases, and the wire features, with +the sets they form. The profiles' facts and `features.json` come from it, and +[`catalog/gen/catalog.json`](catalog/gen/catalog.json) holds it as JSON for +readers without CUE. See [the proposal](https://github.com/ComlineProject/docs/pull/162) +and [core#219](https://github.com/ComlineProject/core/issues/219) for what +will read it. ## The corpus -- [`features.json`](features.json): the wire features, with stable ids: +- [`features.json`](features.json), generated from the catalog: the wire + features, with stable ids: `name-hash`, `handshake`, `envelope`, `framing-datagram`, `framing-datagram2`, `framing-datagram3`, `handshake-comline2`, `framing-jsonrpc`, `stream`, `format-json`, `format-msgpack` and `format-packed`. @@ -310,7 +317,8 @@ define. Rust's are in [`runtime/conformance`](https://github.com/ComlineProject/ about how it encodes, which the version comparison turns into bumps. A format states `struct_fields`, `unknown_field`, `missing_optional`, `enum_variants`, `unknown_variant`, `integer_width` and `output`; a framing states `calls`, -`errors` and `payload`. +`directions`, `errors` and `payload`. The catalog gives each fact's values and +what they mean ([`catalog/facts.cue`](catalog/facts.cue)). `payload` says what a framing needs of a payload, and `output` what a format writes. A framing takes a format when its `payload` is `opaque` (it carries the @@ -353,7 +361,8 @@ profile states it until live cross-runtime calls can prove it (see below). Vendor `features.json`, `vectors/` and `profiles/` at a pinned commit of this repository, and `specs/gen/` too if you want their `$schema` to resolve; a -reader ignores it. `benches/` carries its own schema, so it vendors whole on +reader ignores it. `catalog/gen/catalog.json` holds the names and their +meanings. `benches/` carries its own schema, so it vendors whole on its own. Keep the pinned revision in your repository and check in CI that your copy still matches it. comline-typescript does this in `runtime/test/conformance/`. diff --git a/catalog/encodings.cue b/catalog/encodings.cue new file mode 100644 index 0000000..c85e573 --- /dev/null +++ b/catalog/encodings.cue @@ -0,0 +1,80 @@ +package catalog + +encodings: { + "packed@1": { + kind: "format" + title: "packed" + description: "Fixed-width fields back to back, varint lengths, no tags." + feature: "format-packed" + tags: ["official"] + } + "json@1": { + kind: "format" + title: "JSON" + description: "Values as UTF-8 JSON: a struct as an object." + feature: "format-json" + tags: ["official"] + } + "msgpack@1": { + kind: "format" + title: "MessagePack" + description: "Values as MessagePack: a struct as an array." + feature: "format-msgpack" + tags: ["official"] + } + "jsonrpc@1": { + kind: "framing" + title: "JSON-RPC" + description: "JSON-RPC 2.0 requests and responses." + aliases: ["jsonrpc", "json-rpc", "jsonrpc-2.0"] + frozen_as: "jsonrpc" + feature: "framing-jsonrpc" + tags: ["official"] + } + "datagram@1": { + kind: "framing" + title: "datagram" + description: "Comline's own binary framing of requests and responses." + aliases: ["datagram", "comline.datagram"] + frozen_as: "datagram" + feature: "framing-datagram" + tags: ["official"] + } + "datagram@2": { + kind: "framing" + title: "datagram@2" + description: "The datagram framing with a request/reply kind and a protocol number, for calls both ways and several protocols on one connection." + feature: "framing-datagram2" + tags: ["official"] + } + "datagram@3": { + kind: "framing" + title: "datagram@3" + description: "The datagram framing that addresses a call by its protocol's and function's wire numbers." + feature: "framing-datagram3" + since: "0.4" + tags: ["official"] + } + "comline@1": { + kind: "handshake" + title: "comline@1" + description: "The fixed frame each end sends first: magic, version, IR hash, wire format, framing and capabilities." + feature: "handshake" + tags: ["official"] + } + "comline@2": { + kind: "handshake" + title: "comline@2" + description: "The compatibility-checking handshake: lines, shape hashes and wire numbers instead of an exact IR match." + feature: "handshake-comline2" + since: "0.4" + tags: ["official"] + } + none: { + kind: "handshake" + title: "no handshake" + description: "No frame and nothing checked. Unversioned, since there is nothing to evolve; it establishes neither agreement." + profile: false + tags: ["official"] + } +} diff --git a/catalog/facts.cue b/catalog/facts.cue new file mode 100644 index 0000000..0f4eb65 --- /dev/null +++ b/catalog/facts.cue @@ -0,0 +1,131 @@ +package catalog + +// In the order a profile lists them, which `specs/profile.cue` and its +// generated schema follow. +facts: { + struct_fields: { + applies_to: "format" + title: "struct fields" + description: "How a struct's fields are written." + values: { + by_position: "back to back, in declaration order" + by_name: "each keyed by its name" + } + } + unknown_field: { + applies_to: "format" + title: "unknown field" + description: "What a reader does with a field its type doesn't have." + values: { + rejected: "refuses the value" + ignored: "skips the field" + } + } + missing_optional: { + applies_to: "format" + title: "missing optional" + description: "What a reader does when an optional field isn't there." + values: { + rejected: "refuses the value" + absent: "reads it as absent" + } + } + enum_variants: { + applies_to: "format" + title: "enum variants" + description: "How an enum's variant is written." + values: { + by_name: "by its name" + by_index: "by its position in the declaration" + } + } + unknown_variant: { + applies_to: "format" + title: "unknown variant" + description: "What a reader does with a variant its type doesn't have." + values: { + rejected: "refuses the value" + tolerated: "reads it without failing" + } + } + integer_width: { + applies_to: "format" + title: "integer width" + description: "Whether an integer reads into another width." + values: { + by_value: "any width the value fits" + exact: "only the width it was written as" + } + } + output: { + applies_to: "format" + title: "output" + description: "The grammar a format writes in, which a framing's `payload` must take." + since: "0.7" + values: { + bytes: "bytes in no grammar a framing reads" + json: "JSON text" + } + } + calls: { + applies_to: "framing" + title: "calls" + description: "How a request names the function it calls." + values: { + by_index: "by the function's position" + by_name: "by the function's name" + by_id: "by the protocol's and function's wire numbers" + } + } + directions: { + applies_to: "framing" + title: "directions" + description: "Whether a frame says if it's a request or a response." + since: "0.10" + values: { + both: "it does, so calls can go both ways on one transport" + one: "it doesn't" + } + } + errors: { + applies_to: "framing" + title: "errors" + description: "How an error response names its error." + values: { + by_ordinal: "by its position in the declaration" + by_name: "by its name" + } + } + payload: { + applies_to: "framing" + title: "payload" + description: "What a framing needs of a payload, which a format's `output` must fit." + since: "0.7" + values: { + opaque: "any bytes, carried as they are" + json: "JSON text, written into the frame" + } + } + schema_agreement: { + applies_to: "handshake" + title: "schema agreement" + description: "Whether the handshake checks both ends were built from compatible schemas." + since: "0.2" + establishes: "schema_agreement" + values: { + "true": "it does, and refuses a peer that wasn't" + "false": "it doesn't" + } + } + encoding_agreement: { + applies_to: "handshake" + title: "encoding agreement" + description: "Whether the handshake checks both ends use the same format and framing." + since: "0.2" + establishes: "encoding_agreement" + values: { + "true": "it does, and refuses a peer that doesn't" + "false": "it doesn't" + } + } +} diff --git a/catalog/features.cue b/catalog/features.cue new file mode 100644 index 0000000..e24fc4f --- /dev/null +++ b/catalog/features.cue @@ -0,0 +1,55 @@ +package catalog + +// In the order `features.json` lists them. +features: { + "name-hash": { + title: "name hash" + description: "FNV-1a 64 of a wire format or framing name, as the handshake carries it" + } + handshake: { + title: "handshake" + description: "the fixed 31-byte frame each side sends first: magic, version, IR hash, wire format, framing, capabilities" + } + envelope: { + title: "envelope" + description: "a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault" + } + "framing-datagram": { + title: "datagram framing" + description: "Comline's own binary framing of requests and responses" + } + "framing-datagram2": { + title: "datagram@2 framing" + description: "the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection" + } + "framing-datagram3": { + title: "datagram@3 framing" + description: "the datagram framing that addresses a call by its protocol's and function's wire numbers, both varints (calls = by_id); replies as datagram@2" + since: "0.4" + } + "handshake-comline2": { + title: "comline@2 handshake" + description: "the compatibility-checking handshake: its hello, answer and verdict messages, and its negotiation steps under every parameter, with every refusal reason" + since: "0.4" + } + stream: { + title: "stream framing" + description: "frames on a byte stream (TCP, a pipe): each behind a u32 little-endian length prefix, refused over the limit, by default 16 MiB (16777216 bytes) in every runtime" + } + "framing-jsonrpc": { + title: "JSON-RPC framing" + description: "JSON-RPC 2.0 requests and responses, a fault as -32603" + } + "format-json": { + title: "JSON wire format" + description: "values as UTF-8 JSON: a struct as an object" + } + "format-msgpack": { + title: "MessagePack wire format" + description: "values as MessagePack: a struct as an array" + } + "format-packed": { + title: "packed wire format" + description: "values as packed@1: fixed-width fields back to back, varint lengths, no tags" + } +} diff --git a/catalog/gen/catalog.json b/catalog/gen/catalog.json new file mode 100644 index 0000000..6c6a477 --- /dev/null +++ b/catalog/gen/catalog.json @@ -0,0 +1,458 @@ +{ + "guarantees": { + "schema_agreement": { + "status": "stable", + "title": "schema agreement", + "description": "Both ends were built from compatible schemas, which the handshake checks.", + "comes_from": "handshake" + }, + "encoding_agreement": { + "status": "stable", + "title": "encoding agreement", + "description": "Both ends use the same format and framing, which the handshake checks.", + "comes_from": "handshake" + }, + "encoding_conformance": { + "status": "stable", + "title": "encoding conformance", + "description": "An implementation encodes what its name promises, proven by conformance vectors.", + "comes_from": "encoding" + }, + "version_honesty": { + "status": "stable", + "title": "version honesty", + "description": "A version bump says what the change breaks, as the version comparison works it out.", + "comes_from": "policy" + }, + "bounded_size": { + "status": "stable", + "title": "bounded size", + "description": "Every value has a maximum size, so a decoder refuses an oversized one before allocating it.", + "comes_from": "type" + } + }, + "facts": { + "struct_fields": { + "applies_to": "format", + "status": "stable", + "title": "struct fields", + "description": "How a struct's fields are written.", + "values": { + "by_position": "back to back, in declaration order", + "by_name": "each keyed by its name" + } + }, + "unknown_field": { + "applies_to": "format", + "status": "stable", + "title": "unknown field", + "description": "What a reader does with a field its type doesn't have.", + "values": { + "rejected": "refuses the value", + "ignored": "skips the field" + } + }, + "missing_optional": { + "applies_to": "format", + "status": "stable", + "title": "missing optional", + "description": "What a reader does when an optional field isn't there.", + "values": { + "rejected": "refuses the value", + "absent": "reads it as absent" + } + }, + "enum_variants": { + "applies_to": "format", + "status": "stable", + "title": "enum variants", + "description": "How an enum's variant is written.", + "values": { + "by_name": "by its name", + "by_index": "by its position in the declaration" + } + }, + "unknown_variant": { + "applies_to": "format", + "status": "stable", + "title": "unknown variant", + "description": "What a reader does with a variant its type doesn't have.", + "values": { + "rejected": "refuses the value", + "tolerated": "reads it without failing" + } + }, + "integer_width": { + "applies_to": "format", + "status": "stable", + "title": "integer width", + "description": "Whether an integer reads into another width.", + "values": { + "by_value": "any width the value fits", + "exact": "only the width it was written as" + } + }, + "output": { + "applies_to": "format", + "status": "stable", + "title": "output", + "description": "The grammar a format writes in, which a framing's `payload` must take.", + "since": "0.7", + "values": { + "bytes": "bytes in no grammar a framing reads", + "json": "JSON text" + } + }, + "calls": { + "applies_to": "framing", + "status": "stable", + "title": "calls", + "description": "How a request names the function it calls.", + "values": { + "by_index": "by the function's position", + "by_name": "by the function's name", + "by_id": "by the protocol's and function's wire numbers" + } + }, + "directions": { + "applies_to": "framing", + "status": "stable", + "title": "directions", + "description": "Whether a frame says if it's a request or a response.", + "since": "0.10", + "values": { + "both": "it does, so calls can go both ways on one transport", + "one": "it doesn't" + } + }, + "errors": { + "applies_to": "framing", + "status": "stable", + "title": "errors", + "description": "How an error response names its error.", + "values": { + "by_ordinal": "by its position in the declaration", + "by_name": "by its name" + } + }, + "payload": { + "applies_to": "framing", + "status": "stable", + "title": "payload", + "description": "What a framing needs of a payload, which a format's `output` must fit.", + "since": "0.7", + "values": { + "opaque": "any bytes, carried as they are", + "json": "JSON text, written into the frame" + } + }, + "schema_agreement": { + "applies_to": "handshake", + "status": "stable", + "title": "schema agreement", + "description": "Whether the handshake checks both ends were built from compatible schemas.", + "since": "0.2", + "establishes": "schema_agreement", + "values": { + "true": "it does, and refuses a peer that wasn't", + "false": "it doesn't" + } + }, + "encoding_agreement": { + "applies_to": "handshake", + "status": "stable", + "title": "encoding agreement", + "description": "Whether the handshake checks both ends use the same format and framing.", + "since": "0.2", + "establishes": "encoding_agreement", + "values": { + "true": "it does, and refuses a peer that doesn't", + "false": "it doesn't" + } + } + }, + "encodings": { + "packed@1": { + "kind": "format", + "status": "stable", + "title": "packed", + "description": "Fixed-width fields back to back, varint lengths, no tags.", + "aliases": [], + "profile": true, + "feature": "format-packed", + "tags": [ + "official" + ] + }, + "json@1": { + "kind": "format", + "status": "stable", + "title": "JSON", + "description": "Values as UTF-8 JSON: a struct as an object.", + "aliases": [], + "profile": true, + "feature": "format-json", + "tags": [ + "official" + ] + }, + "msgpack@1": { + "kind": "format", + "status": "stable", + "title": "MessagePack", + "description": "Values as MessagePack: a struct as an array.", + "aliases": [], + "profile": true, + "feature": "format-msgpack", + "tags": [ + "official" + ] + }, + "jsonrpc@1": { + "kind": "framing", + "status": "stable", + "title": "JSON-RPC", + "description": "JSON-RPC 2.0 requests and responses.", + "aliases": [ + "jsonrpc", + "json-rpc", + "jsonrpc-2.0" + ], + "frozen_as": "jsonrpc", + "profile": true, + "feature": "framing-jsonrpc", + "tags": [ + "official" + ] + }, + "datagram@1": { + "kind": "framing", + "status": "stable", + "title": "datagram", + "description": "Comline's own binary framing of requests and responses.", + "aliases": [ + "datagram", + "comline.datagram" + ], + "frozen_as": "datagram", + "profile": true, + "feature": "framing-datagram", + "tags": [ + "official" + ] + }, + "datagram@2": { + "kind": "framing", + "status": "stable", + "title": "datagram@2", + "description": "The datagram framing with a request/reply kind and a protocol number, for calls both ways and several protocols on one connection.", + "aliases": [], + "profile": true, + "feature": "framing-datagram2", + "tags": [ + "official" + ] + }, + "datagram@3": { + "kind": "framing", + "status": "stable", + "title": "datagram@3", + "description": "The datagram framing that addresses a call by its protocol's and function's wire numbers.", + "aliases": [], + "profile": true, + "feature": "framing-datagram3", + "since": "0.4", + "tags": [ + "official" + ] + }, + "comline@1": { + "kind": "handshake", + "status": "stable", + "title": "comline@1", + "description": "The fixed frame each end sends first: magic, version, IR hash, wire format, framing and capabilities.", + "aliases": [], + "profile": true, + "feature": "handshake", + "tags": [ + "official" + ] + }, + "comline@2": { + "kind": "handshake", + "status": "stable", + "title": "comline@2", + "description": "The compatibility-checking handshake: lines, shape hashes and wire numbers instead of an exact IR match.", + "aliases": [], + "profile": true, + "feature": "handshake-comline2", + "since": "0.4", + "tags": [ + "official" + ] + }, + "none": { + "kind": "handshake", + "status": "stable", + "title": "no handshake", + "description": "No frame and nothing checked. Unversioned, since there is nothing to evolve; it establishes neither agreement.", + "aliases": [], + "profile": false, + "tags": [ + "official" + ] + } + }, + "features": { + "name-hash": { + "status": "stable", + "title": "name hash", + "description": "FNV-1a 64 of a wire format or framing name, as the handshake carries it" + }, + "handshake": { + "status": "stable", + "title": "handshake", + "description": "the fixed 31-byte frame each side sends first: magic, version, IR hash, wire format, framing, capabilities" + }, + "envelope": { + "status": "stable", + "title": "envelope", + "description": "a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault" + }, + "framing-datagram": { + "status": "stable", + "title": "datagram framing", + "description": "Comline's own binary framing of requests and responses" + }, + "framing-datagram2": { + "status": "stable", + "title": "datagram@2 framing", + "description": "the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection" + }, + "framing-datagram3": { + "status": "stable", + "title": "datagram@3 framing", + "description": "the datagram framing that addresses a call by its protocol's and function's wire numbers, both varints (calls = by_id); replies as datagram@2", + "since": "0.4" + }, + "handshake-comline2": { + "status": "stable", + "title": "comline@2 handshake", + "description": "the compatibility-checking handshake: its hello, answer and verdict messages, and its negotiation steps under every parameter, with every refusal reason", + "since": "0.4" + }, + "stream": { + "status": "stable", + "title": "stream framing", + "description": "frames on a byte stream (TCP, a pipe): each behind a u32 little-endian length prefix, refused over the limit, by default 16 MiB (16777216 bytes) in every runtime" + }, + "framing-jsonrpc": { + "status": "stable", + "title": "JSON-RPC framing", + "description": "JSON-RPC 2.0 requests and responses, a fault as -32603" + }, + "format-json": { + "status": "stable", + "title": "JSON wire format", + "description": "values as UTF-8 JSON: a struct as an object" + }, + "format-msgpack": { + "status": "stable", + "title": "MessagePack wire format", + "description": "values as MessagePack: a struct as an array" + }, + "format-packed": { + "status": "stable", + "title": "packed wire format", + "description": "values as packed@1: fixed-width fields back to back, varint lengths, no tags" + } + }, + "sets": { + "guarantees": [ + "schema_agreement", + "encoding_agreement", + "encoding_conformance", + "version_honesty", + "bounded_size" + ], + "formats": [ + "packed@1", + "json@1", + "msgpack@1" + ], + "framings": [ + "jsonrpc@1", + "datagram@1", + "datagram@2", + "datagram@3" + ], + "handshakes": [ + "comline@1", + "comline@2", + "none" + ], + "official": [ + "packed@1", + "json@1", + "msgpack@1", + "jsonrpc@1", + "datagram@1", + "datagram@2", + "datagram@3", + "comline@1", + "comline@2", + "none" + ], + "profiled": [ + "packed@1", + "json@1", + "msgpack@1", + "jsonrpc@1", + "datagram@1", + "datagram@2", + "datagram@3", + "comline@1", + "comline@2" + ], + "format_facts": [ + "struct_fields", + "unknown_field", + "missing_optional", + "enum_variants", + "unknown_variant", + "integer_width", + "output" + ], + "framing_facts": [ + "calls", + "directions", + "errors", + "payload" + ], + "handshake_facts": [ + "schema_agreement", + "encoding_agreement" + ], + "features": [ + "name-hash", + "handshake", + "envelope", + "framing-datagram", + "framing-datagram2", + "framing-datagram3", + "handshake-comline2", + "stream", + "framing-jsonrpc", + "format-json", + "format-msgpack", + "format-packed" + ], + "aliases": { + "jsonrpc": "jsonrpc@1", + "json-rpc": "jsonrpc@1", + "jsonrpc-2.0": "jsonrpc@1", + "datagram": "datagram@1", + "comline.datagram": "datagram@1" + } + } +} diff --git a/catalog/guarantees.cue b/catalog/guarantees.cue new file mode 100644 index 0000000..c9af68f --- /dev/null +++ b/catalog/guarantees.cue @@ -0,0 +1,29 @@ +package catalog + +guarantees: { + schema_agreement: { + title: "schema agreement" + description: "Both ends were built from compatible schemas, which the handshake checks." + comes_from: "handshake" + } + encoding_agreement: { + title: "encoding agreement" + description: "Both ends use the same format and framing, which the handshake checks." + comes_from: "handshake" + } + encoding_conformance: { + title: "encoding conformance" + description: "An implementation encodes what its name promises, proven by conformance vectors." + comes_from: "encoding" + } + version_honesty: { + title: "version honesty" + description: "A version bump says what the change breaks, as the version comparison works it out." + comes_from: "policy" + } + bounded_size: { + title: "bounded size" + description: "Every value has a maximum size, so a decoder refuses an oversized one before allocating it." + comes_from: "type" + } +} diff --git a/catalog/readme.md b/catalog/readme.md new file mode 100644 index 0000000..c8b6abd --- /dev/null +++ b/catalog/readme.md @@ -0,0 +1,50 @@ +# The catalog + +Every name that more than one Comline program has to agree on, defined once. +Each entry has a stable id, a title and a description, and where they apply +a value domain, aliases, a status and plain references to other entries. +Sets of names are queries over the entries, so they can't drift from them. + +| Kind | Ids | File | +| --- | --- | --- | +| Guarantees | `schema_agreement`, `bounded_size`, ... | [`guarantees.cue`](guarantees.cue) | +| Facts, with their values | `struct_fields`, `calls`, `schema_agreement`, ... | [`facts.cue`](facts.cue) | +| Official encodings, with their aliases | `packed@1`, `jsonrpc@1`, `comline@2`, `none`, ... | [`encodings.cue`](encodings.cue) | +| Wire features | `name-hash`, `framing-datagram3`, ... | [`features.cue`](features.cue) | +| Sets | `formats`, `framing_facts`, `aliases`, ... | [`sets.cue`](sets.cue) | + +What an entry may say, and the rules across entries, are in +[`schema.cue`](schema.cue): + +- An id is a published name, so it never changes. A rename adds an entry + and deprecates the old one (`status: "deprecated"`, `replaced_by`). +- An alias resolves to exactly one encoding, and is never another + encoding's name. Where core freezes another spelling than the wire name + (`frozen_as`: `jsonrpc`, `datagram`), that spelling is an alias too. +- A reference (`establishes`, `feature`) names an entry that exists. +- A handshake's facts are booleans. +- `since` is the spec version that introduced an entry, where the changelog + names one. + +## What reads it + +- [`specs/profile.cue`](../specs/profile.cue) takes each kind's facts and + their values from here, so a profile is held to the catalog's domains. +- [`features.json`](../features.json) is generated from `features`. +- [`gen/catalog.json`](gen/catalog.json) is the whole catalog as JSON, for + readers without CUE: the compiler, the runtimes, generators, the language + server, the Observatory and the docs. +- `specs/check.sh` checks that `profiles/` holds exactly the encodings with a + profile, each of its encoding's kind, and `vectors/` exactly the features. + +A third party's encoding isn't an entry here: its package declares it in +`config.idp`, in the same shape, and the same rules judge it. + +## Changing it + +Edit the `.cue` files, then run `specs/check.sh` to regenerate +`features.json` and `gen/catalog.json`. CI runs `specs/check.sh --check`, +which fails when either is stale. + +Order matters where it shows: facts are listed in the order profiles list +them, and features in the order `features.json` does. diff --git a/catalog/schema.cue b/catalog/schema.cue new file mode 100644 index 0000000..ac9282c --- /dev/null +++ b/catalog/schema.cue @@ -0,0 +1,96 @@ +package catalog + +import "list" + +// What every entry may say about where it stands. `since` is the spec +// version that introduced it, where the changelog names one. +#Lifecycle: { + status: *"stable" | "experimental" | "deprecated" + replaced_by?: string + since?: =~"^[0-9]+\\.[0-9]+$" + if status == "deprecated" { + replaced_by!: string + } +} + +// A guarantee Comline gives between peers, which a protocol can give up +// with `@unverified`. Its id is the name written in `accept_unverified`, +// diagnostics and generated docs. +#Guarantee: close({ + #Lifecycle + title!: string + description!: string + // What gives it up: the handshake, the format and framing, a handshake + // policy, or a type. + comes_from!: "handshake" | "encoding" | "policy" | "type" +}) + +// A fact an encoding's profile states, with every value it may take and +// what each means. A handshake's facts are booleans: `true` establishes the +// guarantee named in `establishes`. +#Fact: close({ + #Lifecycle + applies_to!: "format" | "framing" | "handshake" + title!: string + description!: string + values!: {[string]: string} + establishes?: or([for id, _ in guarantees {id}]) +}) + +// An official encoding, by its wire name: the spelling the handshake +// hashes, which never changes. +#Encoding: close({ + #Lifecycle + kind!: "format" | "framing" | "handshake" + title!: string + description!: string + // Other spellings a schema may write, resolved to this name. + aliases: [...string] + // The spelling core freezes into the IR, where it isn't the wire name: + // the one used before the names were versioned. + frozen_as?: string + // Whether `profiles/.json` states its facts. + profile: bool | *true + // The feature whose vectors prove it. + feature?: or([for id, _ in features {id}]) + tags: [...string] +}) + +// A wire feature: a set of vector cases, `vectors/.json`. +#Feature: close({ + #Lifecycle + title!: string + description!: string +}) + +// Ids, and nothing else, in each kind: a pattern constraint alone would let +// a label that doesn't match through unchecked. +#SnakeId: "^[a-z][a-z_]*$" +#EncodingId: "^[a-z0-9._-]+(@[0-9]+)?$" +#FeatureId: "^[a-z0-9-]+$" + +guarantees: {[=~#SnakeId]: #Guarantee, [!~#SnakeId]: _|_} +facts: {[=~#SnakeId]: #Fact, [!~#SnakeId]: _|_} +encodings: {[=~#EncodingId]: #Encoding, [!~#EncodingId]: _|_} +features: {[=~#FeatureId]: #Feature, [!~#FeatureId]: _|_} + +// A handshake's facts are booleans. +_boolean_handshake_facts: [for id, f in facts if f.applies_to == "handshake" if [for v, _ in f.values {v}] != ["true", "false"] {id}] & [] + +// An alias names one encoding, and is never another's wire name or frozen +// spelling. A frozen spelling is also an alias. +_spellings: { + for id, e in encodings for a in e.aliases {(a): id} +} +_aliases_are_unique: list.UniqueItems() & [for id, e in encodings for a in e.aliases {a}] +_aliases_are_not_names: [for a, _ in _spellings if encodings[a] != _|_ {a}] & [] +_frozen_is_an_alias: [for id, e in encodings if e.frozen_as != _|_ if !list.Contains(e.aliases, e.frozen_as) {id}] & [] + +// An official profile, `profiles/.json`: of an encoding the catalog +// gives a profile, and of that encoding's kind. `specs/profile.cue` says the +// rest. +#OfficialProfile: { + name!: or(sets.profiled) + kind!: encodings[name].kind + ... +} diff --git a/catalog/sets.cue b/catalog/sets.cue new file mode 100644 index 0000000..6932211 --- /dev/null +++ b/catalog/sets.cue @@ -0,0 +1,35 @@ +package catalog + +// Each set is a query over the entries, so it can't drift from them. A +// label that names a top-level field is quoted, so it doesn't shadow it. +sets: { + "guarantees": [for id, _ in guarantees {id}] + formats: [for id, e in encodings if e.kind == "format" {id}] + framings: [for id, e in encodings if e.kind == "framing" {id}] + handshakes: [for id, e in encodings if e.kind == "handshake" {id}] + official: [for id, e in encodings for t in e.tags if t == "official" {id}] + // The encodings with a profile: `profiles/.json` exists for each. + profiled: [for id, e in encodings if e.profile {id}] + format_facts: [for id, f in facts if f.applies_to == "format" {id}] + framing_facts: [for id, f in facts if f.applies_to == "framing" {id}] + handshake_facts: [for id, f in facts if f.applies_to == "handshake" {id}] + // Every feature: `vectors/.json` exists for each. + "features": [for id, _ in features {id}] + // Every alias, to the wire name it resolves to. + aliases: _spellings +} + +// What `catalog/gen/catalog.json` holds, for readers without CUE. +catalog: { + "guarantees": guarantees + "facts": facts + "encodings": encodings + "features": features + "sets": sets +} + +// What `features.json` holds, in the shape `specs/features.cue` gives it. +features_json: { + $schema: "specs/gen/features.schema.json" + "features": [for id, f in features {"id": id, title: f.title, description: f.description}] +} diff --git a/cue.mod/module.cue b/cue.mod/module.cue new file mode 100644 index 0000000..d41722a --- /dev/null +++ b/cue.mod/module.cue @@ -0,0 +1,4 @@ +module: "github.com/comlineproject/specification@v0" +language: { + version: "v0.17.1" +} diff --git a/features.json b/features.json index 7327bab..54063cc 100644 --- a/features.json +++ b/features.json @@ -1,17 +1,65 @@ { - "$schema": "specs/gen/features.schema.json", - "features": [ - { "id": "name-hash", "title": "name hash", "description": "FNV-1a 64 of a wire format or framing name, as the handshake carries it" }, - { "id": "handshake", "title": "handshake", "description": "the fixed 31-byte frame each side sends first: magic, version, IR hash, wire format, framing, capabilities" }, - { "id": "envelope", "title": "envelope", "description": "a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault" }, - { "id": "framing-datagram", "title": "datagram framing", "description": "Comline's own binary framing of requests and responses" }, - { "id": "framing-datagram2", "title": "datagram@2 framing", "description": "the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection" }, - { "id": "framing-datagram3", "title": "datagram@3 framing", "description": "the datagram framing that addresses a call by its protocol's and function's wire numbers, both varints (calls = by_id); replies as datagram@2" }, - { "id": "handshake-comline2", "title": "comline@2 handshake", "description": "the compatibility-checking handshake: its hello, answer and verdict messages, and its negotiation steps under every parameter, with every refusal reason" }, - { "id": "stream", "title": "stream framing", "description": "frames on a byte stream (TCP, a pipe): each behind a u32 little-endian length prefix, refused over the limit, by default 16 MiB (16777216 bytes) in every runtime" }, - { "id": "framing-jsonrpc", "title": "JSON-RPC framing", "description": "JSON-RPC 2.0 requests and responses, a fault as -32603" }, - { "id": "format-json", "title": "JSON wire format", "description": "values as UTF-8 JSON: a struct as an object" }, - { "id": "format-msgpack", "title": "MessagePack wire format", "description": "values as MessagePack: a struct as an array" }, - { "id": "format-packed", "title": "packed wire format", "description": "values as packed@1: fixed-width fields back to back, varint lengths, no tags" } - ] + "$schema": "specs/gen/features.schema.json", + "features": [ + { + "id": "name-hash", + "title": "name hash", + "description": "FNV-1a 64 of a wire format or framing name, as the handshake carries it" + }, + { + "id": "handshake", + "title": "handshake", + "description": "the fixed 31-byte frame each side sends first: magic, version, IR hash, wire format, framing, capabilities" + }, + { + "id": "envelope", + "title": "envelope", + "description": "a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault" + }, + { + "id": "framing-datagram", + "title": "datagram framing", + "description": "Comline's own binary framing of requests and responses" + }, + { + "id": "framing-datagram2", + "title": "datagram@2 framing", + "description": "the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection" + }, + { + "id": "framing-datagram3", + "title": "datagram@3 framing", + "description": "the datagram framing that addresses a call by its protocol's and function's wire numbers, both varints (calls = by_id); replies as datagram@2" + }, + { + "id": "handshake-comline2", + "title": "comline@2 handshake", + "description": "the compatibility-checking handshake: its hello, answer and verdict messages, and its negotiation steps under every parameter, with every refusal reason" + }, + { + "id": "stream", + "title": "stream framing", + "description": "frames on a byte stream (TCP, a pipe): each behind a u32 little-endian length prefix, refused over the limit, by default 16 MiB (16777216 bytes) in every runtime" + }, + { + "id": "framing-jsonrpc", + "title": "JSON-RPC framing", + "description": "JSON-RPC 2.0 requests and responses, a fault as -32603" + }, + { + "id": "format-json", + "title": "JSON wire format", + "description": "values as UTF-8 JSON: a struct as an object" + }, + { + "id": "format-msgpack", + "title": "MessagePack wire format", + "description": "values as MessagePack: a struct as an array" + }, + { + "id": "format-packed", + "title": "packed wire format", + "description": "values as packed@1: fixed-width fields back to back, varint lengths, no tags" + } + ] } diff --git a/specs/check.sh b/specs/check.sh index dca6757..4981511 100755 --- a/specs/check.sh +++ b/specs/check.sh @@ -1,8 +1,9 @@ #!/bin/sh # Generates gen/*.schema.json (and benches/bench.schema.json, which ships with -# the benches so a vendored copy of them is whole) from the specs and vets -# every corpus file against them. `--check` fails instead of writing when the -# generated schemas are stale. +# the benches so a vendored copy of them is whole) from the specs, and +# features.json and catalog/gen/catalog.json from the catalog, and vets every +# corpus file against them. `--check` fails instead of writing when a +# generated file is stale. set -eu cd "$(dirname "$0")" @@ -40,6 +41,46 @@ generate Report gen/report.schema.json generate Support gen/support.schema.json generate Vectors gen/vectors.schema.json +# What the catalog exports: features.json, and the whole catalog as JSON for +# readers without CUE. +export_json() { # + if $check; then + if ! cue export ../catalog -e "$1" | cmp -s - "$2"; then + echo "$2 is out of date: run specs/check.sh" >&2 + failed=1 + fi + else + cue export ../catalog -e "$1" -o "$2" -f + fi +} + +cue vet -c ../catalog || failed=1 +export_json features_json ../features.json +export_json catalog ../catalog/gen/catalog.json + +# profiles/ and vectors/ hold exactly what the catalog names. +same_names() { # + if [ "$(printf '%s\n' "$2" | sort)" != "$(printf '%s\n' "$3" | sort)" ]; then + echo "$1 differ from the catalog's: expected" $2 "- found" $3 >&2 + failed=1 + fi +} +names_in() { # + for file in "$1"/*.json; do basename "$file" .json; done +} +same_names profiles "$(cue export ../catalog -e 'strings.Join(sets.profiled, "\n")' --out text)" "$(names_in ../profiles)" +same_names vectors "$(cue export ../catalog -e 'strings.Join(sets.features, "\n")' --out text)" "$(names_in ../vectors)" +for file in ../profiles/*.json; do + if ! cue vet -c -d '#OfficialProfile' ../catalog "json:" "$file"; then + echo "$file isn't the profile the catalog gives its encoding" >&2 + failed=1 + fi + if [ "$(cue export "json:" "$file" -e name --out text)" != "$(basename "$file" .json)" ]; then + echo "$file names another encoding" >&2 + failed=1 + fi +done + # Not the schema the benches ship with. vet Bench $(ls ../benches/*.json | grep -v '/bench\.schema\.json$') vet Features ../features.json diff --git a/specs/cue.mod/module.cue b/specs/cue.mod/module.cue deleted file mode 100644 index a15554c..0000000 --- a/specs/cue.mod/module.cue +++ /dev/null @@ -1,4 +0,0 @@ -module: "cue.example" -language: { - version: "v0.17.1" -} diff --git a/specs/gen/profile.schema.json b/specs/gen/profile.schema.json index ef00545..c7d0187 100644 --- a/specs/gen/profile.schema.json +++ b/specs/gen/profile.schema.json @@ -373,7 +373,7 @@ "minItems": 1 } }, - "description": "`profiles/.json`: facts about how an encoding encodes, each naming\nthe vector cases that prove it.", + "description": "`profiles/.json`: facts about how an encoding encodes, each naming\nthe vector cases that prove it. Which facts each kind states, and the\nvalues each may take, are the catalog's (`catalog/facts.cue`).", "anyOf": [ { "$ref": "#/$defs/FormatProfile" diff --git a/specs/profile.cue b/specs/profile.cue index 38c2a69..7c92626 100644 --- a/specs/profile.cue +++ b/specs/profile.cue @@ -1,7 +1,10 @@ package conformance +import "github.com/comlineproject/specification/catalog" + // `profiles/.json`: facts about how an encoding encodes, each naming -// the vector cases that prove it. +// the vector cases that prove it. Which facts each kind states, and the +// values each may take, are the catalog's (`catalog/facts.cue`). #Profile: #FormatProfile | #FramingProfile | #HandshakeProfile // A format states every fact. @@ -10,13 +13,9 @@ package conformance name!: #ProfileName kind!: "format" facts!: close({ - struct_fields!: close({value!: "by_position" | "by_name", proven_by!: #ProvenBy}) - unknown_field!: close({value!: "rejected" | "ignored", proven_by!: #ProvenBy}) - missing_optional!: close({value!: "rejected" | "absent", proven_by!: #ProvenBy}) - enum_variants!: close({value!: "by_name" | "by_index", proven_by!: #ProvenBy}) - unknown_variant!: close({value!: "rejected" | "tolerated", proven_by!: #ProvenBy}) - integer_width!: close({value!: "by_value" | "exact", proven_by!: #ProvenBy}) - output!: close({value!: "bytes" | "json", proven_by!: #ProvenBy}) + for id, fact in catalog.facts if fact.applies_to == "format" { + (id)!: close({value!: or([for v, _ in fact.values {v}]), proven_by!: #ProvenBy}) + } }) }) @@ -26,10 +25,9 @@ package conformance name!: #ProfileName kind!: "framing" facts!: close({ - calls!: close({value!: "by_index" | "by_name" | "by_id", proven_by!: #ProvenBy}) - directions!: close({value!: "both" | "one", proven_by!: #ProvenBy}) - errors!: close({value!: "by_ordinal" | "by_name", proven_by!: #ProvenBy}) - payload!: close({value!: "opaque" | "json", proven_by!: #ProvenBy}) + for id, fact in catalog.facts if fact.applies_to == "framing" { + (id)!: close({value!: or([for v, _ in fact.values {v}]), proven_by!: #ProvenBy}) + } }) }) @@ -39,8 +37,9 @@ package conformance name!: #ProfileName kind!: "handshake" facts!: close({ - schema_agreement?: #Agreement - encoding_agreement?: #Agreement + for id, fact in catalog.facts if fact.applies_to == "handshake" { + (id)?: #Agreement + } }) }) diff --git a/specs/readme.md b/specs/readme.md index a089653..ae41769 100644 --- a/specs/readme.md +++ b/specs/readme.md @@ -15,14 +15,19 @@ runtime that vendors `benches/` alone has it. | `#Vectors` | `vectors/*.json` | [`vectors.cue`](vectors.cue) | Values and types (`#Value`, `#Type`) are in [`types.cue`](types.cue), and what -the others share in [`common.cue`](common.cue). `$schema` is optional in every +the others share in [`common.cue`](common.cue). Which facts a profile states, +and their values, come from [the catalog](../catalog/readme.md), which the +specs import: the CUE module (`cue.mod/`) is the repository's root, so run +`cue` anywhere inside it. `$schema` is optional in every file: a third party's needn't have one. ## Building and verifying -`./check.sh` regenerates `gen/` and `benches/bench.schema.json` and vets every -file, and CI runs `./check.sh --check`, which fails instead when a generated -schema is stale. It needs +`./check.sh` regenerates `gen/` and `benches/bench.schema.json` from the +specs, and `../features.json` and `../catalog/gen/catalog.json` from the +catalog, and vets every file. It also checks that `profiles/` and `vectors/` +hold exactly what the catalog names. CI runs `./check.sh --check`, which fails +instead when a generated file is stale. It needs [`cue`](https://cuelang.org) at the version `.github/workflows/check.yml` pins.