Schema IR for the WorkTable designer: read a declaration as data, diff it, emit DSL and UML - #86
Closed
pathscale wants to merge 6 commits into
Closed
Schema IR for the WorkTable designer: read a declaration as data, diff it, emit DSL and UML#86pathscale wants to merge 6 commits into
pathscale wants to merge 6 commits into
Conversation
added 6 commits
September 1, 2026 19:57
A schema is written down once, in a `worktable!` invocation, and the parser that understands it lived in `worktable_codegen`, which is `proc-macro = true`. A proc-macro crate can export nothing but macros, so every type describing a schema — the columns, the primary key, the indexes, the queries — was unreachable from any other crate however public it was declared. `mod common` was not public at that crate's root either. So anything wanting to *read* a declaration had two options: re-implement the grammar and drift from it, or do without. A diagram, a migration tool, a documentation generator and an editor all want to read one. `lib.rs` has carried `// TODO: Refactor this codegen stuff because it's now too strange.` `model` and `parser` move to `worktable_dsl`, a plain library. Nothing in them changed; the dependencies are the five they already used, none added, none dropped. `worktable_codegen` now depends on it, so there is one grammar rather than a copy that can disagree with the compiler about what a schema means. `name_generator` stays in codegen. It invents Rust identifiers for generated code, which is not the schema language, and generators here define inherent `impl`s on `WorktableNameGenerator` — the orphan rule allows that only in the crate owning the type. I had it in the extracted crate first and the compiler made the same argument the design does. `crate::common::` still resolves, through a thin module that re-exports the new crate, so the 127 paths across 67 files are untouched and the diff stays a move rather than a sweep. An integration test reads a declaration from outside, which is the claim worth pinning: it compiles as its own crate, so it stops building if this ever becomes a proc-macro crate again. It also records a property no caller existed to depend on before. `Columns::columns_map` is a `std::collections::HashMap`, whose iteration order Rust randomises per process; two runs of the same input gave `["answered", "project_id", "id"]` and `["project_id", "answered", "id"]`. The macro never cared, and the parser's own tests collect it into another `HashMap` and assert membership, so nothing noticed. A consumer rendering columns in that order draws a different table every run. `field_positions` already carries the declaration order and is the field to sort by; the test asserts that, and says so, so the next consumer learns it here rather than by shipping the bug.
`pub use worktable_dsl::{Parser, *};` names `Parser` and then re-exports it
again through the glob, and rustc reports the glob as unused: nothing reaches
the shim that way, because every caller goes through `crate::common::model::`
or `crate::common::parser::`.
That is only a warning locally, which is why it survived the extraction. CI
runs `cargo clippy --workspace --all-targets -- -D warnings`, where it is a
build failure.
`worktable_dsl` can parse a declaration, which is half of what a designer needs. The other half is holding one, comparing it, storing it, and writing it back out as text the compiler accepts. `crate::model` cannot do any of that: it is built out of `Ident` and `TokenStream`, which is right for a thing whose job is to become Rust code and wrong for everything else. An `Ident` cannot be serialised or constructed outside a proc-macro context without a `Span::call_site` that lies about where it came from, and a `TokenStream` is not `PartialEq`, so two schemas cannot even be asked whether they differ. That is the one question a migration planner exists to answer. `schema::Schema` is the same declaration with the compiler's concerns removed: `String` for `Ident`, ordered `Vec`s for `HashMap`s, no spans. It derives `PartialEq` so schemas can be diffed, and under an optional `serde` feature it derives `Serialize`, so one can be stored next to the data it describes and read back by a process that never saw the Rust type. The feature is off by default because `worktable_codegen` depends on this crate and is a proc macro: every WorkTable user compiles it for the host before anything else in their build, and they should not pay for a derive macro that serves consumers who are not the compiler. The approach is additive. Nothing in the parser or the generators changed, and the model is untouched apart from three `cfg_attr` derives on plain enums the IR reuses rather than duplicates. Inverting the parser to produce plain data directly would have been a 61-file edit across 13k lines of generators with no test proving the output was unchanged. Two emitters. `to_dsl` renders the declaration body back to text, which is what makes a drawing editable: read, change, write, compile. `to_mermaid` renders UML class notation, chosen because it is text, so it is diffable and needs no rendering dependency, and because it renders anywhere Markdown does. Columns are attributes carrying their markers, queries are operations, and the partition key is a note rather than an attribute because it is stored once per partition and no query can name it. The schema language has no foreign keys, so `infer_relations` guesses links from a single stated naming rule and returns what it guessed; `schemas_to_mermaid` draws those as dependencies rather than associations, because a dashed arrow is the honest notation for a link the declaration does not make. Ordering is a guarantee here, not an accident. `columns_map` is a `HashMap`, whose iteration order Rust randomises, so a consumer walking it draws a different table on every run; `field_positions` records the declaration order and is what the IR sorts by. The query maps have no such field, so those are sorted by name, which is at least stable, and a test says which is which. The round trip is checked against the repository rather than against a fixture. `tests/round_trip.rs` finds all 128 `worktable!` invocations in the tree, sets aside the 12 that are `macro_rules!` templates full of metavariables, and asserts `parse(emit(parse(x))) == parse(x)` for the remaining 116. Those were written by people not thinking about this crate, which makes them a better corpus than anything written here. `codegen` adds the claim this crate cannot make about itself: that emitted text is a declaration the macro accepts. That check has to live on the near side of the proc-macro boundary. Three things the corpus turned up. The emitter writes no comma after `delete`, `in_place` or `config` blocks. `parse_updates` consumes one and those three do not, so a comma there arrives at a dispatch loop as a `,` token and dies as "Unexpected identifier". Omitting it is the only form all of them accept. Expanding one declaration twice does not produce one program. Several generators iterate `columns_map` directly to emit an ordered construct, the `RowFields` and `AvaiableTypes` enums among them, so the variant order differs between two expansions in one process and can differ between two compilations of the same source. `generator_determinism` records it with the evidence and is ignored rather than deleted: the fix changes the generated code of every table and deserves reviewing on its own. It is also why `emitted_declarations` can only assert that an emitted declaration expands, not that it generates identical code.
With a schema as data on both sides, the decision a version mismatch forces can
be computed instead of hand-written. `Diff::between` says what changed, `Cost`
says what applying it costs, and `transforms_required` says which parts a person
still has to write, which are the parts that need intent rather than mechanism.
The cost model is about links, not about fields. A row is addressed by a
`Link { page_id, offset, length }`, and every index holds links, so the question
that decides what a change costs is not how many columns moved but whether a row
is still where it was. Any change to the archived layout invalidates every link
in the table at once and there is no cheaper answer than writing every row
somewhere else. A change to an index invalidates nothing: the rows have not
moved and the index can be rebuilt from them. Hence three tiers and a fourth for
the changes no diff can settle.
That last tier is the point of the exercise. A changed primary key, a changed
partition key, a renamed table and a flipped `persist` are not expensive, they
are underdetermined, and the useful thing a planner can do is say so rather than
guess. The routing key is the clearest case: it is not in the row, so which
partition a row belongs to cannot be recomputed from the row, only from where
the row already is.
The planner invents a value only when there is exactly one it could be. Widening
a column to `optional` has one answer. Narrowing it does not, and neither does
adding a required column or changing a type, so each of those comes back as a
`TransformRequest` naming the column and why. A rename is reported as a drop and
an add, because nothing in a declaration distinguishes it from a deletion beside
an unrelated addition, and guessing by type equality would be wrong exactly when
it mattered.
Three things worth knowing that the tests state as claims. A version bump on its
own costs nothing, which is what keeps bumping cheap enough to be habitual.
Reordering columns is a layout change, because declaration order is the row
struct's field order: it is the change most likely to be made by accident and
least likely to look like one. And a schema that changed without a version bump
is still detected, at the cost of comparing two small structs and reading no
rows, which is the middle branch of the load state machine.
`plan` lifts the same comparison to a set of tables, matching by name because
that is how spaces are matched on disk. A dropped table is `NeedsIntent` rather
than free: whether to delete data is a decision, not a consequence of a
declaration.
A compiled binary could not say what schema it was built against. The
information was there at expansion time and then thrown away, so a migration
planner had no "declared" side to compare against what is on disk, and a
designer could not draw a diagram of an application whose source it did not
have.
Every generated table now carries a `<NAME>_SCHEMA` const, named after
`<NAME>_VERSION` because it answers the question next to it: the version says
which schema, and this says what that schema is.
The stored form is the DSL text rather than a serialised structure. It needs no
format decision, keeps serde out of the dependency graph of every user's build,
is legible in a hex dump, and is read back by the same parser that read the
original. It is also, being a declaration, exactly what regenerates an old table
type, which is the hand-maintained `version_tables: { 1 => v1::UserV1WorkTable }`
that makes migrations something people put off. `dsl/tests/round_trip.rs` holds
the property this rests on against all 116 declarations in this repository, and
the tests here check the emitted const against the declaration it came from and
that the macro accepts it back.
In-memory tables get it too. A designer reading a crate wants every table, and
the const costs a string either way.
Two details worth the words. The second parse runs at the end of `expand` rather
than the start, so this function's diagnostics stay the ones a bad declaration
produces: both parses reject the same inputs, but only one of them knows to say
that a separate `attributes` section is not part of the 1.0 grammar. And the
const is `allow(dead_code)`, because a `worktable!` inside a function body puts
it inside that body, where nothing refers to it and `-D warnings` would fail a
user's build over a const they never asked for.
A designer opening a project, a documentation generator and a migration tool
comparing a checkout against a running database all start from the same problem:
a schema is written inside a `worktable!` invocation somewhere in a crate, and
there is no index of where.
`declarations_in_source` walks tokens rather than `syn`'s item tree, because an
invocation inside a function body is not an item and real code puts them there.
An item walk would quietly miss a table, and the caller would never learn a table
existed to be missed. Both delimiter forms are accepted: the repository uses
`worktable!( .. )` 83 times and `worktable! { .. }` 45 times, and a reader that
took only one would be wrong about the language.
The return is not a `Vec<Schema>`. Some invocations are not declarations: a
`macro_rules!` body writing `name: $name, ... using $backend` is a template whose
metavariables stand for text that exists only after the outer macro expands, and
counting those as failures would be wrong. Everything else that fails to parse is
reported with the text that failed, because a designer that silently drops a
table the compiler accepts is worse than one that says it could not read it.
The corpus round-trip test now goes through this, so it is also the evidence that
the scanner finds what is there: 116 declarations read, 12 templates set aside,
nothing rejected, across the whole repository.
Merged
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pushed so this is not lost and the next agent can build on it. Not fully validated — treat the later commits as a starting point, not finished work.
What is here
Six commits. The first is the same extraction that is already up as #79.
8ba799aworktable_dsl(also in #79)dfe8ef66c76a6e)fdb3f9587214e28f0dad0ede0f47New module
dsl/src/schema/:mod.rs,emit_dsl.rs,emit_uml.rs,diff.rs,scan.rs. Tests indsl/tests/:schema.rs,round_trip.rs,diff.rs,readable_from_outside.rs.Overlap with #79 — read before merging
This branch contains #79's commit, rebased onto master. Do not merge both as-is.
Recommended order: merge #79 first (it is green and mergeable), then rebase this branch onto master and drop the two duplicated commits. The alternative — merging this and closing #79 — also works, but #79 has the CI history proving the extraction is clean.
What the IR is
worktable_dsl::schema, an additive module. The parser, the model and the generators are unchanged apart from threecfg_attrserde derives on plain enums the IR reuses rather than duplicating.Schemaand friends:StringforIdent, orderedVecwhere the parser usedHashMap, no spans,PartialEq, andSerialize/Deserializebehind an off-by-defaultserdefeature. Off by default becauseworktable_codegenis a proc macro that every user compiles for the host before anything else in their build.Schema::parse(&str)/Schema::from_tokens— the macro's own dispatch, parse-only. ASchemacan hold a declaration the macro would refuse to expand, which is what a designer needs while somebody is still typing.to_dsl()/to_macro_invocation()— the declaration back as text.to_mermaid()/schemas_to_mermaid()/infer_relations()— UML class notation.Validation that exists
parse(emit(parse(x))) == parse(x)is checked against the repository, not a fixture:dsl/tests/round_trip.rsfinds all 128worktable!invocations in the tree, sets aside the 12 that aremacro_rules!templates full of metavariables, and round-trips the remaining 116.dsl/tests/schema.rshas 15 claims, one per test.codegen/src/worktable/mod.rs::emitted_declarationsadds the one claim this crate cannot make about itself — that emitted text is a declaration the macro accepts — because that check has to live on the near side of the proc-macro boundary.The last two commits (
8f0dad0,ede0f47) are the least validated. Run CI before trusting them.Known unresolved
infer_relationsguesses from one stated naming rule and draws dependencies (..>), never associations. A designer must not present a guess as a fact.HashMap;field_positionsrecovers declaration order. Anything new reading columns must go through it.