perf: marshal the C read path's dicts in C (numpy C-API), ~1.5x reads

[jameskermode]

What

Move the C read path's dict marshalling into the _extxyz extension. After the C reader parses a frame, it previously handed the data back to Python by walking the DictEntry linked list one field at a time through ctypes (c_to_py_dict) — ~1.09M ctypes.cast + 737k copy.copy across a 76k-frame read, pure boundary overhead that dominates files with many small frames.

This adds a CPython C-API entry point (_extxyz.read_frame, in libextxyz/pyext.c) that builds the info/arrays dicts of numpy arrays / scalars directly in C and frees the C dicts, replacing the per-node Python loop.

Why it's safe

• Bit-identical to the old path. tests/test_marshal_parity.py reads each fixture through both backends (and both use_regex values) and asserts equal keys, dtypes, shapes, types, and bit-identical floats. Verified additionally on all 76,346 frames of a real training set: 0 mismatches.
• Graceful fallback. cextxyz.read_frame_dicts dispatches to the C path only when present; otherwise it uses the preserved read_frame_dicts_ctypes. EXTXYZ_LEGACY_MARSHAL=1 forces the legacy path.
• C core untouched. Zero diff to extxyz.c / the extxyz.h ABI. pyext.c is compiled only into the _extxyz Python extension — never the standalone libextxyz shared library or the C/Fortran (fextxyz/QUIP)/Julia consumers.
• No leaks. leaks over 12k frames across success/EOF/scattered-string/error paths: 0 leaks, 0 bytes.

Build / packaging

• numpy is now a build-time dependency (already a runtime dep). The numpy include is resolved non-fatally in meson.build: a numpy-less standalone C/Fortran/Julia build still configures and just uses the ctypes path. Verified both a numpy-present and a numpy-less meson setup/compile.
• Windows: a second module-def (_extxyz_pyext.def) exports PyInit__extxyz so the module is importable; the numpy-less build keeps the original .def (no unresolved-export link error).
• CI now asserts every built wheel and the source build actually shipped the C path (tools/assert_c_read_path.py), so a silent ctypes fallback fails the build instead of shipping quietly slow.

Measured (76,346 frames, ~27 atoms/frame)

metric	before	after	speedup
dict-level parse (read_dicts)	3.6 s	2.3 s	~1.5×
full ASE read (format='cextxyz')	4.8 s	3.3 s	1.48×

Most visible on many-small-frame files / rich comment lines, where per-frame overhead — not per-atom parsing — dominates; on the large single-frame Cu benchmark the effect is small.

Notes for reviewers

• The remaining gap to hand-written parsers (e.g. chemfiles) is now mostly the libcleri comment-line grammar — a possible follow-up (hand-written C key=value scanner), intentionally out of scope here.
• Benchmark/verification scripts included under benchmarks/ (bench_mad.py, scope_comment.py, verify_marshal.py).

🤖 Generated with Claude Code