Feat/cross language serialization - #92
Merged
Merged
Conversation
Adds an Architecture page describing how one domain model reaches many languages: who owns .proto files, who implements a runtime, and how the two meet at the registry. Four diagrams carry the parts that are hard to convey in prose -- the mirrored per-language structure, the seven AnyData arms (including why `fallback` is the only one that reconstructs an object), the recursive dispatch tree, and the schema distribution model. Mermaid needed enabling: superfences had no custom_fence for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Downstream generators need the schemas themselves, not only the bindings this
repository happens to emit, so `create-proto-bundle` zips them for release
upload. It is rooted at the include path, keeping the `compas_pb/generated/`
prefix intact -- a flat archive would break every import of message.proto.
TypeScript joins the generated languages via @bufbuild/protoc-gen-es, installed
into a version-keyed cache the same way protoc is. protoc emits no TypeScript
natively, so PROTO_PLUGIN_LANGUAGES maps a language to its plugin flag and
javascript or go can be added the same way.
Plugin-backed assets are pinned by plugin version rather than protoc version,
since the plugin is what shapes the generated API.
The task identity (package name, generated folder) is now configurable, so any
package owning .proto files can reuse this pipeline; antikythera already
imports generate_proto_classes from here.
Two fixes along the way:
- the stale-asset cleanup globbed dist/*.zip while assets are written to
dist/proto/*.zip, so it matched nothing and old zips survived into the
release upload
- an unsupported target language printed a warning and then ran protoc with
an empty --_out flag; it now raises
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reusing these tasks from antikythera surfaced it: compas_pb configures base_folder as a Path, antikythera as a str from os.path.dirname, and create_class_assets did `base_dir / "dist"` on whatever it was given. Same for proto_folder. Both are normalized now, so a consumer can configure either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What a compas_pb_rust or compas_pb_go has to do to interoperate, what changes elsewhere when a language is added, and which parts of the wire format are easy to get subtly wrong. The traps section is the part worth having written down: each of the four produced a real bug while compas_pb_ts was being built, and none of them fail loudly. An integer arriving as a float, bytes surfacing as the literal string "base64:...", a domain object sent as dict_value and arriving as a bare dictionary because only the fallback arm reconstructs, and short-name type URL matching that works right up until two packages register the same class name. Also records the rules that came out of the same work: consume schemas from a pinned release artifact rather than vendoring them, resolve shared compas_pb types to the runtime package's copy rather than a second generated one, and test against bytes Python produced, since a round trip passes even when both directions are wrong in the same way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Opens with why anyone would want this rather than with the contract, and cuts roughly a third of the words. The five failure modes are folded into one section, since what matters about each is the symptom and the rule, not the reasoning that led there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The failure-modes section read as a list of bugs we once hit, which is not useful to someone starting a runtime. The rules in it are, though -- a runtime that does not know bytes are base64-tagged, or that only `fallback` rebuilds an object, looks fine and quietly corrupts data. They are now four short lines next to the arms they belong to, phrased as rules rather than as history. The same went for the anecdote about consumers scraping git. Reworded throughout for readers who code but are not full-time developers: shorter sentences, fewer clauses, plainer words. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same four steps as antikythera had, now the shared action. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends compas_pb’s release asset generation so downstream runtimes (e.g. TypeScript) can consume pinned .proto schemas and language-specific generated bindings, and documents the cross-language serialization architecture to guide additional runtime implementations.
Changes:
- Add support for plugin-based protoc generation (initially TypeScript via
protoc-gen-es) and adjust asset naming/versioning accordingly. - Publish a
.proto-only bundle alongside per-language generated binding zip assets. - Add architecture/runtime-implementation documentation and enable Mermaid diagrams in the docs site.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
tasks.py |
Exposes the new proto-bundling task in the Invoke task collection. |
src/compas_pb/invocations.py |
Adds TypeScript plugin setup, supports plugin-backed generation, and produces .proto + per-language binding zip assets. |
mkdocs.yml |
Enables Mermaid fenced blocks and adds new architecture/runtime docs pages to nav. |
docs/architecture.md |
New architecture spec for cross-language runtime/domain-owner responsibilities and wire format. |
docs/implementing-a-runtime.md |
New guide/checklist for implementing a runtime in a new language. |
.gitignore |
Ignores transient per-language generated output folders under src/compas_pb/generated/. |
.github/workflows/build.yml |
Switches asset creation/upload to the shared release-assets action. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| zip_path = dist_dir / f"{_package_name(ctx)}-proto.zip" | ||
| zip_path.unlink(missing_ok=True) | ||
|
|
||
| proto_root = Path(ctx.proto_include_paths[0]) |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of change is this?
Checklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.CHANGELOG.mdfile in theUnreleasedsection under the most fitting heading (e.g.Added,Changed,Removed).invoke test).invoke lint).compas.datastructures.Mesh.