Skip to content

Run the specification's compiler conformance cases (#224) - #243

Merged
Kinflou merged 3 commits into
masterfrom
conformance/compiler-cases
Nov 11, 2026
Merged

Kinflou merged 3 commits into
masterfrom
conformance/compiler-cases

Conversation

@Kinflou

@Kinflou Kinflou commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Runs the specification's compiler conformance cases (ComlineProject/specification#5, core#224) against core. Each case is a schema graph given as data, together with the outcome any compiler must reproduce.

Spec pin. tests/spec/ vendors the specification at 0685006, the merge commit of specification#5, and comline-catalog is pinned to the same commit, so the test that REV equals the catalog rev holds.

What it does

tests/package/compiler_cases.rs handles each of the 42 cases:

  • it writes the case's packages side by side in a temp directory;
  • it builds the case's build package with compile_package, which resolves dependencies by path;
  • it compares the outcome with the case's expect.

The comparison covers:

  • whether the build succeeded;
  • each protocol's frozen given-up set;
  • who acknowledged each guarantee (the protocol or the package) and why;
  • the warnings, by code and severity;
  • for a failed build, its coded errors (the [code] markers in the rendered error) and any error_contains text.

Every mismatch is reported with its case id before the test fails. The runner has a CORE_DISAGREES list for cases that core gets wrong; it is empty, so every case must pass.

The test runs in the --features deps jobs, because the dependency cases need dependency resolution.

What running it found

39 of 42 cases matched on the first run. The 3 recursion cases were wrong in the spec, not in core:

  • They recursed through optional next: Node.
  • Core deliberately doesn't let an optional field end a cycle yet (OPTIONAL_ENDS_A_CYCLE), because the Rust generator doesn't box optional fields. So it refused the type with validation::struct-cycle before checking bounds.
  • specification#5 9e590a2 changes them to recurse through kids: Node[max 4], which leaves depth as the only unbounded measure.

No core bugs were found, including in the three-package chain cases.

Kinflou added 2 commits November 11, 2026 10:43
#224)

Vendors compiler/cases/*.json beside the size vectors and runs each: the
package graph is laid out in a temp dir and built with compile_package, then
the frozen given-up sets, acknowledgements, diagnostics and errors compared.
Pins comline-catalog to the unmerged specification#5 head.
@Kinflou Kinflou changed the title compiler cases: run the specification's conformance cases against core (#224) Run the specification's compiler conformance cases (#224) Nov 11, 2026
@Kinflou
Kinflou marked this pull request as ready for review November 11, 2026 11:58
@Kinflou
Kinflou merged commit eda2699 into master Nov 11, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant