Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/reports-branch.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ latest verification reports of `main`, which the
from this branch's history:

- `typescript.bench.json`: instruction counts of the shared benchmarks
(ComlineProject/runtime's `runtime/conformance/benches/`), from
(ComlineProject/specification's `benches/`), from
`.github/workflows/bench.yml` on every push to `main`, in two tiers per
benchmark: `instructions` (`node --no-opt`) and `instructions-opt` (the
optimizing tier on, after a warm-up), which is the one the Observatory's
Expand Down
11 changes: 6 additions & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,10 @@ jobs:
with:
node-version: 20
# The Rust reference the property tests (test/prop/) compare against,
# built at the commit the vectors are vendored from. Without it they
# built at the runtime commit the core is vendored from. Without it they
# fail in CI rather than skip.
- id: reference
run: echo "rev=$(cat test/conformance/VECTORS_REV)" >> "$GITHUB_OUTPUT"
run: echo "rev=$(cat core/CORE_REV)" >> "$GITHUB_OUTPUT"
- uses: actions/cache@v4
id: reference-cache
with:
Expand All @@ -66,8 +66,9 @@ jobs:
# The generated fixtures, against the built runtime as a consumer sees
# it, under the strictest flags and erasable syntax only.
- run: npm run typecheck:generated
# The wire conformance vectors are vendored from ComlineProject/runtime
# at the commit in VECTORS_REV; this fails if the copy was edited.
# The wire conformance vectors are vendored from
# ComlineProject/specification at the commit in SPEC_REV; this fails if
# the copy was edited.
- run: scripts/vendor-conformance.sh --check
# The core (core/: the wasm32 build of comline-runtime-ffi and its JS
# glue) is rebuilt from source at CORE_REV, with the runtime's pinned
Expand Down Expand Up @@ -212,7 +213,7 @@ jobs:
# The Rust reference the property tests compare against, built on the
# platform under test (cached per OS and architecture).
- id: reference
run: echo "rev=$(cat test/conformance/VECTORS_REV)" >> "$GITHUB_OUTPUT"
run: echo "rev=$(cat core/CORE_REV)" >> "$GITHUB_OUTPUT"
- uses: actions/cache@v5
id: reference-cache
with:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
with:
node-version: 22
- id: reference
run: echo "rev=$(cat test/conformance/VECTORS_REV)" >> "$GITHUB_OUTPUT"
run: echo "rev=$(cat core/CORE_REV)" >> "$GITHUB_OUTPUT"
- uses: actions/cache@v6
id: reference-cache
with:
Expand Down Expand Up @@ -86,7 +86,7 @@ jobs:
log="$RUNNER_TEMP/properties.log"
run_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
{
echo "The nightly property tests against \`comline-reference\` (VECTORS_REV \`${{ steps.reference.outputs.rev }}\`, commit ${{ github.sha }}) failed: $run_url"
echo "The nightly property tests against \`comline-reference\` (CORE_REV \`${{ steps.reference.outputs.rev }}\`, commit ${{ github.sha }}) failed: $run_url"
echo
echo "A divergence in no class \`known-divergences.json\` lists, with its shrunk counterexample. Replay it with \`FC_SEED\` and \`FC_PATH\` from the report."
echo
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/reports.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ jobs:
# The Rust reference the property tests compare against, installed as
# ci.yml does (same cache key, same rev).
- id: reference
run: echo "rev=$(cat runtime/test/conformance/VECTORS_REV)" >> "$GITHUB_OUTPUT"
run: echo "rev=$(cat runtime/core/CORE_REV)" >> "$GITHUB_OUTPUT"
- uses: actions/cache@v4
id: reference-cache
with:
Expand Down
17 changes: 9 additions & 8 deletions runtime/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@ resolves before writing) must now copy it.

## Wire conformance

The wire contract is checked against the byte-exact vectors that
`ComlineProject/runtime` keeps in `runtime/conformance/`, so a TypeScript
The wire contract is checked against the byte-exact vectors of the
[Comline specification](https://github.com/ComlineProject/specification), so a TypeScript
peer and a Rust peer generated from the same schema negotiate the same frame
and speak the same request and response bytes.

Expand All @@ -68,7 +68,7 @@ and speak the same request and response bytes.
[`conformance-report.json`](conformance-report.json) with the cases proving
it. The docs' language reference is rendered from the report.
- The vectors and encoding profiles are vendored in `test/conformance/` at the
commit in `VECTORS_REV`. CI checks that the copy is unedited.
specification commit in `SPEC_REV`. CI checks that the copy is unedited.
- `test/conformance-server.ts` serves the conformance stdio ops protocol (the
one `comline-reference` speaks), over the same ops, so `comline conformance`
can drive this runtime. Its `version` reply lists `json@1` and `packed@1`,
Expand Down Expand Up @@ -108,11 +108,12 @@ core can't load and for custom framings. Behaviour is the same either way.

The build is reproducible (the runtime's pinned toolchain, `--locked`, paths
remapped, a fixed `SOURCE_DATE_EPOCH`), so CI rebuilds it from source and
fails unless every byte matches. `CORE_REV` and `VECTORS_REV` move together:
the script refuses to vendor a core at another commit than the vectors.
fails unless every byte matches. `CORE_REV` and `SPEC_REV` move together:
the script refuses to vendor a core whose runtime vendors another
specification commit (its `runtime/conformance/spec/REV`) than `SPEC_REV`.

```sh
scripts/vendor-conformance.sh <rev> # first, the vectors at the same commit
scripts/vendor-conformance.sh <spec> # first, the spec the runtime at <rev> vendors
scripts/vendor-core.sh <rev> # then the core
scripts/vendor-core.sh --check # rebuild CORE_REV, compare the bytes
```
Expand Down Expand Up @@ -160,11 +161,11 @@ so lengths cross varint boundaries.

### Running it

Build the reference at the commit the vectors are vendored from, and point
Build the reference at the runtime commit the core is vendored from, and point
`COMLINE_REFERENCE` at it (or put `comline-reference` on `PATH`):

```sh
# in a checkout of ComlineProject/runtime, at test/conformance/VECTORS_REV
# in a checkout of ComlineProject/runtime, at core/CORE_REV
cargo build --release -p comline-reference

# here
Expand Down
4 changes: 2 additions & 2 deletions runtime/bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ params and result a JSON string <bytes> long.
## Shared benchmarks

`bench/shared.mjs` runs the benchmarks every runtime measures, from the
definitions vendored in `test/conformance/benches/` (ComlineProject/runtime's
`runtime/conformance/benches/`, pinned by `VECTORS_REV`), under their ids:
definitions vendored in `test/conformance/benches/` (ComlineProject/specification's
`benches/`, pinned by `SPEC_REV`), under their ids:
`format/<encode|decode>/<shape>/<format>`,
`framing/<encode|decode>/<framing>/<size>`, `handshake/<encode|decode|check>`
and `connection/step`. Each measures what generated code runs (a codec with
Expand Down
4 changes: 2 additions & 2 deletions runtime/bench/shared.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
// file, You can obtain one at https://mozilla.org/MPL/2.0/.

// The shared benchmarks: the work every runtime measures, from the vendored
// definitions in test/conformance/benches/ (ComlineProject/runtime's
// runtime/conformance/benches/), under the ids its README gives. Each id maps
// definitions in test/conformance/benches/ (ComlineProject/specification's
// benches/), under the ids its README gives. Each id maps
// to one iteration's work, everything it reads built here, outside it.
// `count.mjs` runs them by id; `node bench/shared.mjs` lists the ids this
// runtime measures, each with its iterations. Left out: msgpack@1 (not shipped) and `comline2/*` (the
Expand Down
2 changes: 1 addition & 1 deletion runtime/core/CORE_REV
Original file line number Diff line number Diff line change
@@ -1 +1 @@
ebfb792d8b3bee069b51a13b2b6a2f204f931602
0dffa7a81504df6c6913f8f427f29e4a8a441239
2 changes: 1 addition & 1 deletion runtime/core/SHA256SUMS
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
4ad1bc81e7fa8ce443e3fe9dc1ff413abdc90efdfd7cb9b8997e515d9027a5bc comline_runtime_ffi.wasm
10f05e39296578eb32fc345e6e97e94dff30f00dd48b41f5c83757375da10208 comline_runtime_ffi.wasm
db8f8870b5a3e05da2c24378379346dd1629de6c459a5ccfaf5dfdda377ee819 comline.mjs
f08ea7adc65102f2d9e9dab5ea82cef6d168373bc088f3886017a40c368315cb comline.d.mts
Binary file modified runtime/core/comline_runtime_ffi.wasm
Binary file not shown.
28 changes: 15 additions & 13 deletions runtime/scripts/vendor-conformance.sh
Original file line number Diff line number Diff line change
@@ -1,30 +1,31 @@
#!/usr/bin/env bash
# Vendors the runtime wire conformance corpus (features.json, profiles/, vectors/ and
# the shared benchmark definitions, benches/)
# from ComlineProject/runtime into test/conformance/, at a pinned commit.
# Vendors the Comline specification's conformance corpus (features.json,
# profiles/, vectors/, the shared benchmark definitions in benches/, and
# VERSION) from ComlineProject/specification into test/conformance/, at a
# pinned commit.
#
# scripts/vendor-conformance.sh <rev> # vendor <rev> and pin it in VECTORS_REV
# scripts/vendor-conformance.sh --check # fail if the copy differs from VECTORS_REV's
# scripts/vendor-conformance.sh <rev> # vendor <rev> and pin it in SPEC_REV
# scripts/vendor-conformance.sh --check # fail if the copy differs from SPEC_REV's
set -euo pipefail

here="$(cd "$(dirname "$0")/.." && pwd)"
dest="$here/test/conformance"
repo="https://github.com/ComlineProject/runtime"
repo="https://github.com/ComlineProject/specification"

if [[ "${1:-}" == "--check" ]]; then
rev="$(cat "$dest/VECTORS_REV")"
rev="$(cat "$dest/SPEC_REV")"
else
rev="${1:?usage: vendor-conformance.sh <rev> | --check}"
fi

tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
git clone --quiet --filter=blob:none --no-checkout "$repo" "$tmp/runtime"
git -C "$tmp/runtime" checkout --quiet "$rev" -- runtime/conformance
corpus="$tmp/runtime/runtime/conformance"
git clone --quiet --no-checkout "$repo" "$tmp/specification"
git -C "$tmp/specification" checkout --quiet "$rev" -- vectors profiles benches features.json VERSION
corpus="$tmp/specification"

if [[ "${1:-}" == "--check" ]]; then
if diff -r "$corpus/vectors" "$dest/vectors" && diff -r "$corpus/profiles" "$dest/profiles" && diff -r "$corpus/benches" "$dest/benches" && diff "$corpus/features.json" "$dest/features.json"; then
if diff -r "$corpus/vectors" "$dest/vectors" && diff -r "$corpus/profiles" "$dest/profiles" && diff -r "$corpus/benches" "$dest/benches" && diff "$corpus/features.json" "$dest/features.json" && diff "$corpus/VERSION" "$dest/VERSION"; then
echo "test/conformance matches $repo@$rev"
else
echo "test/conformance differs from $repo@$rev: re-vendor with scripts/vendor-conformance.sh <rev>" >&2
Expand All @@ -38,6 +39,7 @@ cp -r "$corpus/vectors" "$dest/vectors"
cp -r "$corpus/profiles" "$dest/profiles"
cp -r "$corpus/benches" "$dest/benches"
cp "$corpus/features.json" "$dest/features.json"
git -C "$tmp/runtime" rev-parse "$rev" > "$dest/VECTORS_REV"
echo "vendored $repo@$(cat "$dest/VECTORS_REV")"
cp "$corpus/VERSION" "$dest/VERSION"
git -C "$tmp/specification" rev-parse "$rev^{commit}" > "$dest/SPEC_REV"
echo "vendored $repo@$(cat "$dest/SPEC_REV")"
echo "Now: CONFORMANCE_BLESS=1 npm test, and review conformance-report.json"
21 changes: 14 additions & 7 deletions runtime/scripts/vendor-core.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@
# remapped and a fixed SOURCE_DATE_EPOCH, so `--check` can rebuild from source
# and compare bytes rather than trust a committed binary.
#
# CORE_REV and test/conformance/VECTORS_REV move together: the core is proven
# by the vectors of the same commit. Re-vendor both at the same rev
# (scripts/vendor-conformance.sh <rev> first).
# CORE_REV and test/conformance/SPEC_REV move together: the core is proven by
# the specification its runtime vendors (runtime/conformance/spec/REV), so
# SPEC_REV must be that revision (scripts/vendor-conformance.sh <it> first).
set -euo pipefail

here="$(cd "$(dirname "$0")/.." && pwd)"
dest="$here/core"
repo="https://github.com/ComlineProject/runtime"
vectors_rev="$(cat "$here/test/conformance/VECTORS_REV")"
spec_rev="$(cat "$here/test/conformance/SPEC_REV")"
files=(comline_runtime_ffi.wasm comline.mjs comline.d.mts)

if [[ "${1:-}" == "--check" ]]; then
Expand All @@ -34,9 +34,16 @@ git clone --quiet "$repo" "$tmp/runtime"
git -C "$tmp/runtime" checkout --quiet --detach "$rev"
rev="$(git -C "$tmp/runtime" rev-parse HEAD)"

if [[ "$rev" != "$vectors_rev" ]]; then
echo "core rev $rev differs from VECTORS_REV $vectors_rev: they move together;" >&2
echo "run scripts/vendor-conformance.sh $rev first" >&2
runtime_spec="$tmp/runtime/runtime/conformance/spec/REV"
if [[ ! -f "$runtime_spec" ]]; then
echo "$repo@$rev vendors no specification (runtime/conformance/spec/REV):" >&2
echo "the core can only be vendored at a runtime revision that does" >&2
exit 1
fi
runtime_spec_rev="$(cat "$runtime_spec")"
if [[ "$runtime_spec_rev" != "$spec_rev" ]]; then
echo "core rev $rev is proven by specification $runtime_spec_rev, but SPEC_REV is $spec_rev;" >&2
echo "run scripts/vendor-conformance.sh $runtime_spec_rev first" >&2
exit 1
fi

Expand Down
7 changes: 3 additions & 4 deletions runtime/test/conformance.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,14 @@

/**
* Wire conformance: the runtime against the byte-exact vectors of
* `ComlineProject/runtime`'s `runtime/conformance/`, vendored in
* `test/conformance/` at the commit in `VECTORS_REV`.
* `ComlineProject/specification`, vendored in `test/conformance/` at the
* commit in `SPEC_REV`.
*
* Every claim in `conformance-support.json` must hold against the vectors:
* a supported feature passes all its cases, an unsupported one reports each
* as not implemented. The result, with the cases proving each claim, is
* `conformance-report.json`; `CONFORMANCE_BLESS=1 npm test` rewrites it.
* The vectors' format is described in the runtime repository's
* `runtime/conformance/README.md`.
* The vectors' format is described in the specification's README.
*/

import assert from "node:assert/strict";
Expand Down
1 change: 1 addition & 0 deletions runtime/test/conformance/SPEC_REV
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3583aaa321aafc74a06e91cc059f4f71e4b30895
1 change: 0 additions & 1 deletion runtime/test/conformance/VECTORS_REV

This file was deleted.

1 change: 1 addition & 0 deletions runtime/test/conformance/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.10
6 changes: 3 additions & 3 deletions runtime/test/reference.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
* line per line; its README describes it.
*
* The binary is `COMLINE_REFERENCE` (a path), or else `comline-reference` on
* `PATH`. Build it at the commit in `test/conformance/VECTORS_REV`, so it
* answers the way the vendored vectors do.
* `PATH`. Build it at the runtime commit in `core/CORE_REV`, which is proven
* by the vendored vectors, so it answers the way they do.
*/

import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process";
Expand Down Expand Up @@ -94,7 +94,7 @@ export function unavailable(): string | undefined {
? `COMLINE_REFERENCE is ${path}, which isn't an executable`
: "comline-reference isn't on PATH and COMLINE_REFERENCE isn't set";
return (
`${why}. Build it from ComlineProject/runtime at test/conformance/VECTORS_REV ` +
`${why}. Build it from ComlineProject/runtime at core/CORE_REV ` +
"(cargo build --release -p comline-reference) and set COMLINE_REFERENCE to the binary"
);
}
Expand Down
4 changes: 2 additions & 2 deletions runtime/test/values.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
* The conformance corpus's typed values and reader types, and how they map
* to what generated TypeScript holds and declares. Shared by the vector
* runner (`conformance.test.ts`) and the differential properties
* (`prop/*.test.ts`). The encoding is described in the runtime repository's
* `runtime/conformance/README.md`.
* (`prop/*.test.ts`). The encoding is described in the README of
* `ComlineProject/specification`.
*/

import type { Layout } from "../src/index.js";
Expand Down
Loading