perf: opt-in first-char dispatch comment-line parser (use_cleri=False)

[jameskermode]

What

Add an opt-in first-char-dispatch comment-line parser, gated behind a new use_cleri flag (default True, so behaviour is unchanged until opted in). use_cleri=False (C backend only) replaces the libcleri grammar walk for the comment line with extxyz_dispatch_parse (libextxyz/extxyz_dispatch.c).

Why

After the per-atom tokenizer (use_regex=False) and the C marshalling (#44), the dominant remaining read cost is the comment line. A series of measured spikes showed it is ~92% libcleri's generic parse-tree machinery (build + re-walk a CST), not regex matching:

• a different parser generator (packcc) tied JIT'd libcleri,
• a faster allocator (arena) bought ~1%,
• a hand-written first-char dispatcher that folds parse+marshal into one pass measured ~1.4–1.8× on the comment line.

So the win can't be had inside pyleri→libcleri; it needs a leaner parser. This ships that as a fast path — the comment-line analog of the per-atom use_regex=False tokenizer the project already ships.

How it stays provable

The dispatcher dispatches each value on its first non-space char and validates token extents with the grammar's own PCRE2 patterns (same accepted token language by construction), building the same DictEntry as tree_to_dict. libcleri is kept as the canonical grammar, the pure-Python pyleri oracle, and the fallback. Provability shifts from "by generation" to "by differential test":

• tests/test_dispatch_parity.py — bit-identical info/arrays + identical accept/reject vs libcleri over a fixture + 300-line randomized corpus.
• a USE_CLERI=false conftest axis + a CI step run the whole C-backend suite through the dispatcher.

Verification

• 0 mismatches on all 76,346 MAD comment lines (bit-identical), fixtures, and reject cases.
• Full suite 131 passed in both modes (use_cleri=True and USE_CLERI=false).
• 0 leaks (incl. the reject path; also fixes free_DataLinkedList to free per-node string content on the error path).
• Full ASE MAD read 3.5 s → 2.5 s (dict-level 2.4 → 1.5 s).

Safety / ABI

use_cleri threads like use_regex (ase-extxyz → iread_dicts → read_frame_dicts → C-API read_frame / ctypes). extxyz_read_ll_opts gains a trailing int use_cleri; ABI-safe — Fortran/Julia bind the 7-arg extxyz_read_ll, which passes use_cleri=1 (unchanged). The dispatcher is pure C + pcre2, compiled into all targets; regexes compiled+JIT'd once (eager at import, atexit free; symbols added to both .defs for Windows).

Benchmarks (README)

New "Comment-line parser (use_cleri=False)" table — the win is per-frame, so it grows as frames shrink:

atoms / frame	dispatch / cleri	full ASE read
2 000	1.07×	1.02×
100	1.34×	1.19×
5	1.77×	1.36×

Read / tokenizer / write tables + both plots also refreshed on current hardware.

Follow-ups (out of scope)

• A +HC hand-coded-token variant (~1.5× more).
• Flipping the default to use_cleri=False once it has soaked.

🤖 Generated with Claude Code