Repository navigation
Full-corpus build, CLI, documentation, and two corrected results - #1
Merged
Merged
Conversation
added 6 commits
September 17, 2026 10:46
Scales the demo to the whole journal slice and makes the repository usable on its own: a scientist can clone it, run one script, and query the graph from a shell or a browser without reading the source. Corpus 84,801 structures, 4,693,914 atomic sites, 687,310 intermolecular contacts, 16.25 M edges. Loaded with LOAD JSONL rather than statement-by-statement Cypher, and committed chronologically as 26 commits, one per publication year, for time travel. Graph model Adds Atom -[:IN_COMPONENT]-> Component. Without it there is no path from a contact between two atoms to the molecules those atoms belong to, which is what a synthon, coformer or polymorph question needs. The assignment already existed in the parser, so the edge is exact. Adds net_dim / net_dim_weak on every Structure: periodic-net dimensionality as the rank of the lattice-translation subgroup generated by cycles. The covalent bonds must be in that calculation, not only the contacts -- a donor and an acceptor are joined through the molecules they belong to, and a quotient graph built from contacts alone is bipartite, acyclic, and reports almost everything as a 0D finite motif. Adds is_involution on every CONTACT edge. An operation that is its own inverse relates exactly two molecules and generates a ring motif; a screw or glide has infinite order and generates a catemer. Both have a two-fold rotation part, so only the composition distinguishes them. This is what makes an R2,2(8) query correct rather than approximately right. Chemistry Weak C-H...A contacts, with their own kind and no inferred variant, so every existing kind='hbond' statistic is unchanged. Inferred donors restricted to N, O and S; other elements are stored as close_contact and excluded from hydrogen-bond statistics. is_centrosymmetric asked of gemmi per space group. Robustness RDKit's bond-order search does not always terminate and is a C++ call no Python watchdog can interrupt, so it runs in a killable child process. Non-finite floats are nulled at serialisation: json.dumps emits a bare NaN, which is not valid JSON, and one occurrence rejects an entire LOAD JSONL. Interfaces crystal.py: facts, a library of named queries, arbitrary Cypher, time travel, CSV and JSON output. The browser studio gains a query library grouped by subject and a Cypher box that paints the canvas from a user's own query. Documentation and packaging README.md for someone deciding whether to use the graph; TUTORIAL.md for the graph model, the dialect's real limits, worked questions, the ingest path and query performance. MIT licence. Development probes and an internal script removed; dependencies reconciled with what the code imports. Tests Two gates execute every query the project ships -- CLI and front end -- against a live server, the latter with a negative control. corpus_facts.py re-derives every figure the README quotes.
A graph on disk is not resident after a server start, and an unloaded graph answers GRAPH_NOT_FOUND rather than returning nothing, which reads as a broken demo. run.sh now loads whatever is present and says plainly what to do when there is nothing. Also loads the cod_contacts_v2 projection, which --ingest built but never loaded, and corrects both documents: the CIFs are not in the repository, so the first run must be --ingest.
The front-end query gate extracted 12 of the ~127 Cypher strings in the studio
and reported "all shipped queries execute cleanly". Two causes: "${" was tested
before the interpolation table ran, discarding every non-trivial query and
leaving all ten substitution patterns dead; and queries assembled from several
string literals joined by "+" were only ever seen as their first fragment.
Extraction is now targeted at the `cypher:` fields and single-literal helper
calls, and -- more importantly -- the gate asserts that a set of required schema
elements (CONTACT, IN_COMPONENT, HAS_FRAGMENT, net_dim, is_involution, Snapshot,
IN_SPACE_GROUP) appears in what it extracted, and fails when one does not. A
scraper that silently stops matching now fails instead of reporting success over
an empty set. Coverage goes from 12 queries to 76.
With the gate working it immediately found two broken queries in the shipped
studio: `min(s.year)` and `max(s.year)`, neither of which this dialect
implements. Both were wrapped in `.catch(() => 0)` beside a comment saying min
and max do not exist -- so they failed on every run and the fallback covered it.
Removed.
HAS_FRAGMENT was massively over-emitted. `_edge_exists` scanned only the last
400 edges, but Component nodes are deduplicated corpus-wide, so the same
component is reached again thousands of edges later: acetic acid carried 23
parallel edges to `carboxylic_acid`. Any query joining Fragment on both ends of
a contact multiplied by (matches on side 1 x matches on side 2). The guard is
now an exact set.
validate_chemistry.py printed "RESULT: PASS" on an empty corpus -- every element
was skipped, `ok` was never cleared, and it exited 0 having analysed nothing. It
now refuses to report on zero atoms, and all three validators fail cleanly with
no data. None of them could be run as documented anyway: they import `ingest`
with no path bootstrap.
solvents.py had a duplicate dictionary key: "C3 O" mapped to both propanol and
acetone, and the first was silently discarded. With no located hydrogens that
formula genuinely cannot be assigned, so it is now labelled as ambiguous.
Also: CI (static checks and a front-end build -- the query gates need a live
server with the corpus, which CI cannot provide); ruff scoped to correctness
rules and clean; math.isfinite instead of the NaN self-comparison idiom; a dead
local in periodic.py; loop variables bound explicitly in the dimensionality
closure.
Documentation corrections, all verified against the live corpus: the space-group
worked example returned zero rows because SpaceGroup is keyed by number and
carries P 1 21/n 1 for group 14; only count() and avg() exist, not min/max/sum;
toInteger/toFloat do not work; h_inferred is not indexed; UNWIND does work and
is the replacement for the unsupported IN; twelve edge types, not eleven;
db.history() returns 54 entries against the ledger's 26; the versioned graph
holds no atoms or contacts, so no chemistry statistic rewinds; net_dim is -1 for
52.5% of the corpus and the dimensionality percentages are over the scored
subset; Cl is an acceptor so Cl...Cl halogen bonds are detected; the inferred
population has no angular criterion; there is no atom-level fragment membership,
so the motif queries are candidate generators; the manifest files are build
outputs, not repository contents.
…at runs nothing
This dialect has no bound parameters -- `$name` is `PARSE_ERROR: Not
implemented: Parameters` -- so a value can only reach a query by string
interpolation, and the usual protection against injection is unavailable:
WHERE g.hm_symbol = 'P -1' -> 1
WHERE g.hm_symbol = 'P -1' OR 1=1 -> 214
Every query here interpolates because there is no alternative. That is fine for
a local demo over public CC0 data with no untrusted input, and it is not fine in
a service that accepts input from a user. The tutorial now says so, because the
patterns in this repository are exactly what a reader would lift.
pytest was configured over tests/ and carried as a dev dependency, but tests/
holds executable checks that take a live server and report by exit code -- there
is not one test function. Configuring pytest over a directory with no tests
gives a step that passes permanently while running nothing, which is the trap
the rest of this work has been removing. Config and dependency dropped.
measure_perf.py defaulted --turing-dir to a directory run.sh never creates, so
the disk and CSD-scale figures silently vanished, and a query that timed out was
logged as an ordinary row with a successful exit. It now names the outputs it
could not produce and fails.
Also reconciles the commit count: the ledger holds 26 snapshots, one per
publication year, while db.history() reports more entries because each year
costs an intermediate COMMIT plus the CHANGE SUBMIT.
Adds Atom -[:IN_FRAGMENT]-> Fragment, 764,852 edges. HAS_FRAGMENT says a
molecule contains a functional group somewhere; this says which atoms are in
it. Without it a motif query can only ask "an oxygen of a molecule that has a
carboxylic acid", which in a polyfunctional molecule also matches its alcohols,
esters and nitro oxygens -- spot-checking hits of the old query, roughly one in
six were not carboxyl-carboxyl.
The obstacle was that bond-order perception runs in a child process and the
only representation crossing back was a SMILES string, whose atom order is
canonical rather than original, so matched indices could not be mapped to
crystallographic sites. SMARTS matching now happens in the child, on the
perceived molecule, while the caller's atom order still holds; indices map back
through `instances` to site and uid.
The motif queries become both correct and shorter, since the Component hop is
no longer needed to reach a functional group:
MATCH (f1:Fragment)<-[:IN_FRAGMENT]-(a1:Atom)
-[h:CONTACT]->(a2:Atom)-[:IN_FRAGMENT]->(f2:Fragment)
HAS_FRAGMENT stays where the question really is about molecules -- coformer
discovery asks what co-crystallises with an acid-bearing molecule, not with a
particular oxygen.
Corrected numbers, with the HAS_FRAGMENT deduplication fix also in place:
R2,2(8) dimers 12,820 -> 1,354 mean H...O 1.79 A at 171.7 deg
C(4) catemers 1,686 -> 280
synthon, by edges 62.2 / 37.8 homo/hetero
synthon, by bonds 75.0 / 25.0
The previous claim that correcting the homodimer double-count "reverses the
winner" was an artefact of the duplication, which inflated the two synthons by
different factors. The homodimer wins on either reading; the correction widens
the margin. Documents and studio narrative updated, and the synthon panel now
states that only 110 molecules carry both groups, so these are corpus-wide
propensities rather than a contest within molecules that had a choice.
Also adds a `molecule-atoms` CLI query so IN_COMPONENT keeps query coverage
after the studio stopped traversing it, and scopes the front-end gate's
required-token list to what the studio actually uses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scales the demo to the whole journal slice, makes the repository usable on its own, and corrects two results that were wrong.
Corpus
84,805 structures, 4.69 M atomic sites, 687,048 intermolecular contacts, 17.0 M edges. Loaded with
LOAD JSONL(21 M elements in 55 s) and committed chronologically as 26 commits, one per publication year, for time travel.Two results were wrong
Periodic-net dimensionality reported ~89% of structures as 0D isolated motifs, including textbook amide catemers. The quotient graph was built from contact edges alone — but a donor and an acceptor are joined in the crystal through the molecules they belong to, an edge that is not a contact. Built that way the graph is bipartite with no path longer than one hop: no cycles, rank 0 everywhere. Including the covalent bonds contracts each molecule to a point, which is the standard formulation. Result is now 25.1 / 28.0 / 22.0 / 24.8 % across 0D–3D.
The R²₂(8) dimer query matched every hydrogen bond against itself and returned 2₁-screw catemers. A centrosymmetric dimer is its own symmetry image, so it is one edge in a quotient graph, not the reciprocal pair the query hunted. The discriminator is whether the generating operation is an involution — its own inverse, so it relates exactly two molecules — precomputed as
is_involutionon every contact edge.Graph model
Atom -[:IN_COMPONENT]-> Component— there was no path from a contact between two atoms to the molecules those atoms belong to, so synthon, coformer and polymorph questions were unanswerable rather than merely unwritten.Atom -[:IN_FRAGMENT]-> Fragment, 764,852 edges — records which atoms are in a functional group, not merely which molecules contain one. Without it a motif query means "an oxygen of a molecule that has a COOH somewhere", which also matches that molecule's alcohols, esters and nitro oxygens.net_dim/net_dim_weakandis_involutionprecomputed at ingest.Corrected motif numbers
HAS_FRAGMENTwas over-emitted: the dedup guard scanned only the last 400 edges, butComponentnodes are deduplicated corpus-wide, so one component carried up to 23 parallel edges. Any query joiningFragmenton both ends multiplied by the product.The earlier claim that correcting the homodimer double-count "reverses the winner" was an artefact of that duplication. The homodimer wins on either reading.
Chemistry
Weak C–H···A contacts with their own
kindand no inferred variant, so every existingkind='hbond'statistic is unchanged. Inferred donors restricted to N, O and S — fluorine and chlorine cannot donate a proton — with the rest stored asclose_contact, excluded from every hydrogen-bond statistic.is_centrosymmetricasked of gemmi per space group rather than summed from a hand-picked list.Interfaces
crystal.py:facts, 14 named queries, arbitrary Cypher, time travel, CSV/JSON. The studio gains a query library grouped by subject and a Cypher box that paints the canvas from a user's own query.Tests
Two gates execute the shipped queries against a live server. The front-end gate previously extracted 12 of ~127 and reported success; it now recovers 76 and asserts that required schema elements appear in what it extracted, failing if one does not — a scraper that stops matching can no longer report success over an empty set. Fixing it immediately surfaced two broken queries in the studio (
min/max, which this dialect does not implement).validate_chemistry.pyprintedRESULT: PASSon an empty corpus; all three validators now fail cleanly with no data, and none could be run as documented before. A duplicate dictionary key insolvents.pysilently discarded propanol. CI added.Robustness
RDKit's bond-order search does not always terminate and is a C++ call no Python watchdog can interrupt, so it runs in a killable child process. Non-finite floats are nulled at serialisation —
json.dumpsemits a bareNaN, which is not valid JSON, and one occurrence rejects an entireLOAD JSONL.Documentation
README.mdfor someone deciding whether to use the graph;TUTORIAL.mdfor the model, the dialect's real limits, worked questions, the studio, time travel, ingest and query performance. MIT licence. Every figure is re-derived from the live graph bytests/corpus_facts.py.Known limits are stated rather than buried: π-stacking absent, hydrogen positions not normalised, disorder handled only by an occupancy threshold, InChIKey reaching ~36% of components, no query parameters (so no injection protection — interpolation is the only route), and only
count()andavg()among aggregates.