test(runtime): Zig-native C-ABI tests for lower.zig (issue #14)

docs/codegen.md

@@ -46,11 +46,11 @@ The pipeline lives in `src/shinro/codegen/`; the Zig VM lives in `src/shinro/run
 3. **Interpret** (`interpret`). Replay the graph on real numpy inputs as a
    correctness oracle. If the interpreter's output matches a live
    `NumpyBackend` run to float-exactness, the tracer is sound.
-4. **Lower** (`lower_zig`). Walk the graph and emit Zig — a `src/shinro/runtime/`
-   module exposing a `shinro_step` C-ABI function with baked constants,
-   compiled to a `.so`. The generated graph is written to
-   `src/shinro/runtime/graph_data.zig`; the comptime VM that executes it is
-   `src/shinro/runtime/lower.zig`.
+4. **Lower** (`lower_zig`). Walk the graph and emit Zig — the generated graph is
+   written to `src/shinro/runtime/graph_data.zig`; the comptime VM that executes
+   it is `src/shinro/runtime/lower.zig` (graph-agnostic, instantiated via
+   `Vm(Ctx)`), and `src/shinro/runtime/entry.zig` binds that graph and exposes
+   the `shinro_step` C-ABI function, compiled to a `.so`.
 
 ## Module map
 
@@ -65,7 +65,7 @@ The pipeline lives in `src/shinro/codegen/`; the Zig VM lives in `src/shinro/run
 | `codegen/compose.py` | `compose` — merge per-component graphs into one closed-loop step graph, auto-inserting `reshape`/`clip`. |
 | `codegen/lower_zig.py` | Emit `src/shinro/runtime/graph_data.zig` (the graph as Zig constants) from a composed graph. |
 | `demos/demo_codegen.py` | Runnable demo: traces KF+LQR for the base and cartpole plants, composes, and verifies each stage against a live numpy loop. |
-| `src/shinro/runtime/` (Zig) | `build.zig` (build script), `lower.zig` (comptime-unrolled VM), `linalg.zig` (shared linear-algebra kernels), `graph_data.zig` (generated graph). |
+| `src/shinro/runtime/` (Zig) | `build.zig` (build script), `entry.zig` (deployment entry: binds the graph and exports `shinro_step`), `lower.zig` (graph-agnostic comptime-unrolled VM), `linalg.zig` (shared linear-algebra kernels), `graph_data.zig` (generated graph). |
 | `codegen/recipes.py` | The graph-recipe registry (`@register_graph` / `build_recipe`) + the shipped recipes (`closed_loop_tracking`, `policy_only`) and the two shipped default graphs (`build_base_graph` / `build_mpc_composed_graph`). |
 | `scripts/gen_base.py` / `scripts/gen_mpc.py` | Shims over `codegen/recipes.py` (`make zig-gen` / `make zig-mpc-gen` run them). |
 | `codegen/component_cli.py` | The "does my component trace?" gate (`shinro check` / `shinro trace`, plus the kept `scripts/trace_component.py` shim): inventory of registered components, inferred trace contract (`--list`), and trace + interpret-vs-live oracle check (bit-exact) for any config TOML. |

lab-notes/daily/2026-09-28.md

@@ -104,3 +104,199 @@ Zig linalg suite **48 passed** (was 47); full VM (`zig build`) exits 0.
   `lower.zig`'s `matmul`/`vecmat` call sites still compile against the shipped
   graph.
 - `make lint` is unaffected (Zig runtime is outside ruff/pyrefly's surface).
+
+## Zig-native C-ABI tests for `lower.zig` (issue #14)
+
+Branch: `feat/zig-native-lower-tests` (off `main` @ 9547ada)
+
+### Why
+
+Issue #14: `lower.zig`'s C-ABI `shinro_step` had only (a) Zig unit tests for
+`linalg.zig` and (b) the Python ctypes oracle (`tests/test_zig_lowering.py`,
+which builds a `.so` and drives it via `ctypes`). Nothing drove the VM
+natively. The blocker was that `lower.zig` hard-imports the generated
+`graph_data.zig`, which `build.zig` only supplies through `-Dgraph` on the
+deployed lib module.
+
+### What changed
+
+- `src/shinro/runtime/tests/lower_fixture_graph.zig` (new): hand-authored
+  fixture `graph_data.zig` — a 10-node graph (`inp` x, `inp` r → `cst`@`matmul`
+  → `add` → `clip` → named `out`; `tanh`(r) → state `out`). Mirrors the
+  generated `Op`/`Node` schema.
+- `src/shinro/runtime/tests/lower.zig` (new): two native tests driving
+  `shinro_step` directly — named-output + state-output routing, and cross-tick
+  recurrent state feedback (state output fed back as the next tick's input).
+- `src/shinro/runtime/build.zig`: a new test module rooted at `lower.zig` with
+  `addAnonymousImport("graph_data", tests/lower_fixture_graph.zig)`, exposed to
+  `tests/lower.zig` as the `lower` module; added to the `test` step.
+- `src/shinro/runtime/lower.zig`: `export fn shinro_step` → `pub export fn
+  shinro_step` (Zig-level import visibility; the C symbol is unchanged).
+- `src/shinro/runtime/README.md`: two new rows in the files table + a note on
+  the native test.
+
+### Design decisions
+
+- Chose the fixture-graph route over the comptime-injection refactor
+  (`Vm(comptime G: type)`). The fixture is ~60 lines across four files, leaves
+  `lower.zig` untouched apart from one visibility word, and — critically — is
+  **decoupled from `make zig-gen`**: the deployed graph churns (last-build-wins)
+  while the fixture does not, so a graph regeneration cannot silently invalidate
+  the test.
+- The fixture must carry the **full `Op` enum**: `lower.zig`'s `switch
+  (node.op)` is exhaustive with no `else`, so the enum must stay in sync. That
+  is a deliberate loud coupling — a new VM op fails the fixture compile.
+- Only the per-op tables the *present* ops read need to exist (`clip_lo`/
+  `clip_hi` here); the arms of absent ops are comptime-skipped.
+- Prototyping gotcha: `lower.zig`'s `@import("qp.zig")` is resolved **even when
+  `has_solve_qp = false`** (relative file imports are not lazily elided the way
+  the `solver_meta` anonymous import is). `qp.zig` sits next to `lower.zig`, so
+  the test module finds it — but the test module must stay rooted at
+  `lower.zig` for that relative import to resolve. No OSQP bake is needed.
+
+### Verification
+
+- `zig build test`: **51/51 passed** (48 linalg + 1 emosqp + 2 lower).
+- `make test-zig` (zig-gen → build real `.so` → `zig build test` →
+  `pytest tests/test_zig_lowering.py`): **91 passed, 2 skipped** — the
+  `pub export` change did not disturb the ctypes oracle or the `.so` symbol.
+- Prototype sanity check: a deliberately wrong expected output fails the native
+  test, so it genuinely executes the VM.
+
+### Follow-ups
+
+- The fixture covers the ABI plumbing + a representative op mix; full op
+  coverage stays with the Python oracle (`tests/test_zig_lowering.py`,
+  `tests/test_op_shape_matrix.py`).
+- The comptime-injection refactor remains an option if arbitrary inline Zig
+  fixture graphs are wanted later.
+
+## Split the C-ABI export out of `lower.zig` into `entry.zig`
+
+### Why
+
+`lower.zig` conflated three concerns: the VM, the graph binding
+(`@import("graph_data")` + `solver_meta`/`qp`), and the C-ABI surface
+(`export fn shinro_step`). That coupling is what forced a graph fixture to be
+wired into the *module* for every test graph. Separating the binding/export
+from the VM lets the VM take the graph as a comptime parameter
+(`Vm(Ctx)`), so tests — and, later, a multi-graph fixture table — inject their
+own graph with no per-graph anonymous-import choreography.
+
+### What changed
+
+- `src/shinro/runtime/lower.zig` — now graph-agnostic: `pub fn Vm(comptime Ctx:
+  type) type` wraps the whole VM (`workspace`, `step`, and the four helpers) in
+  a container, with `const g = Ctx.graph;` aliasing the injected graph so the
+  **body is byte-for-byte unchanged** (every `g.nodes` / `g.offsets` reference
+  stands). The `graph_data`/`solver_meta`/`qp` imports and the `export fn` are
+  gone; `sm`/`qp` now come from `Ctx` (only referenced for QP graphs).
+- `src/shinro/runtime/entry.zig` (new) — the deployment entry: the only file
+  that imports `graph_data` (plus `solver_meta`/`qp.zig` when QP), assembles
+  `Ctx`, and declares `pub export fn shinro_step` forwarding to
+  `lower.Vm(Ctx).step`.
+- `src/shinro/runtime/build.zig` — lib module root `lower.zig` → `entry.zig`;
+  the VM test module is now an anonymous-import-free `vm_mod`, and the fixture
+  graph is wired to the *driver* (`tests/lower.zig`) instead.
+- `src/shinro/runtime/tests/lower.zig` — binds the fixture itself
+  (`const Ctx = struct { pub const graph = g; … }`) and calls
+  `lower.Vm(Ctx).step`.
+- `src/shinro/runtime/README.md` + `docs/codegen.md` — table/narrative rows for
+  `entry.zig` and the graph-agnostic VM.
+
+### Verification
+
+- `zig build test`: **51/51 passed** (48 linalg + 1 emosqp + 2 VM).
+- `make test-zig` (real ReleaseFast `.so` + ctypes oracle + pytest): **91
+  passed, 2 skipped** — the shipped KF+LQR oracle, the MPC/`.solve_qp` oracle,
+  and the ONNX/recurrent oracles all still match `interpret()`, so the QP path
+  through `entry.zig`'s conditional `solver_meta`/`qp` wiring is intact.
+- Binary check: built pre-refactor HEAD in a scratch worktree and diffed the
+  `.so`. With `step` marked `inline` (needed so the exported wrapper compiles
+  to exactly the old body), the shipped **ReleaseFast `.so` is byte-identical**
+  to pre-refactor (`sha256 7d42e97b…`) — so production is provably unchanged,
+  not merely behaviorally equivalent. The Debug build differs by 32 B of
+  argument-spill slots on the exported wrapper (debug-info only); behavior is
+  oracle-verified in both modes.
+
+### Follow-ups
+
+- The multi-graph fixture table (one shared `Vm(Ctx)` driver + N generated
+  `(graph, vectors)` pairs) is now a pure build.zig table addition — no VM
+  changes, no per-graph module instances.
+
+## Frozen multi-graph VM fixtures (real HF policies)
+
+### Why
+
+Step 2 of the native-test work: drive `Vm(Ctx)` with a menu of real graph
+types — classical, MLP, and recurrent — generated once and committed, so the
+VM is regression-tested against them without any generation-time churn.
+
+### What changed
+
+- `scripts/gen_lower_fixtures.py` (new): generate-once generator. Per fixture it
+  lowers the composed graph, post-processes the emitted `const_blob_f32` into a
+  raw little-endian `.bin` re-exposed via `@embedFile` + `bytesAsSlice` (an
+  `align(@alignOf(f32))` copy keeps it safe for the kernel's vector weight
+  loads), and emits `interpret()` vectors (8 seeded samples). Deliberately NOT
+  wired into `make zig-gen`/CI.
+- `src/shinro/runtime/tests/graphs/` (new, generated): `kf_lqr`, `toy_lstm`,
+  `go2` (real Unitree Go2 MLP), `drone_gru` (real eco-drone GRU) — each
+  `<name>_graph.zig` + `<name>_weights.bin` + `<name>_data.zig`.
+- `src/shinro/runtime/tests/lower_graph.zig` (new): one shared driver, compiled
+  once per fixture via `lower.Vm(Ctx)`.
+- `build.zig`: a `graph_fixtures` table — one graph+vectors module pair and one
+  test per row. Adding a graph type is one row.
+
+### Why the compact encoding
+
+The real policies lower to big *Zig source* because the f32 weight blob is
+written as hex literals: go2 4.1 MB, drone_gru 1.9 MB. A raw f32 `.bin` +
+`@embedFile` cuts them to 736 KB / 342 KB — ~1.2 MB for all four fixtures vs
+~6 MB. Compile time was never the issue (go2 builds in ~1.7 s); repo size was.
+
+### Verification
+
+- `zig build test`: **55/55** (48 linalg + 1 emosqp + 2 hand fixture + 4 frozen
+  graph fixtures). The 4 new tests passing validates both the compact blob
+  (alignment + values) and the recurrent state plumbing against `interpret()`.
+- `make test-zig`: **91 passed, 2 skipped** — unchanged ctypes oracle.
+- Total fixture payload: ~1.24 MB.
+
+### Follow-ups
+
+- Add a fixture = one `FI` entry in the generator + one `graph_fixtures` row.
+- The fixture header records the source (HF path or recipe); no manifest is
+  committed (it was redundant with the graph table).
+
+### C-ABI coverage for the fixtures
+
+`TestFrozenFixturesCAbi` (tests/test_zig_lowering.py) builds a `.so` per fixture
+and drives the **exported** `shinro_step` symbol via ctypes against the same
+committed vectors — covering the export/port-packing surface the in-process
+`Vm(Ctx).step` oracle bypasses, and the compiled path for the compact
+`@embedFile` graph format. `make test-zig`: **95 passed, 2 skipped** (was 91).
+Self-contained: reads only the committed fixtures (no shinro-bench, no ONNX).
+
+### Native QP fixture + C-ABI routing (issue #14 closure)
+
+Closed the last uncovered VM op (`solve_qp`). The frozen menu gains an `mpc`
+fixture (KF + MPC_LTI, `has_solve_qp = true`, tol 1e-3), and the fixture driver
+now routes through the **exported C-ABI `shinro_step`** instead of
+`Vm(Ctx).step`: each fixture gets its own `entry` module (rooted at `entry.zig`)
+so `graph_data` resolves to that fixture, and the QP one also compiles the baked
+OSQP solver (include paths + C sources + `solver_meta`). `entry.zig`'s
+`shinro_step` became `pub export` so the tests can call it (symbol unchanged).
+
+- `zig build test`: **56/56** (was 55; +the `mpc` fixture). The `.solve_qp` arm
+  now has a Zig-native path (entry → `Vm(Ctx)` → `qp.solve_qp` → the C bake).
+- `make test-zig`: **96 passed, 2 skipped** (was 95/2 — `mpc` also passes through
+  the `.so` via `TestFrozenFixturesCAbi`).
+- The generator now runs `zig fmt` on its output, so regeneration is a no-op
+  against the committed files (only a new fixture shows as a diff).
+
+Pitfall hit while verifying: the fixture `.so` builds land in pytest's `/tmp`;
+a full `/tmp` made `zig build` fail with `DiskQuota`, which the tests surface as
+**skips** (`_build_fixture_so` skips on build failure) — clear scratch dirs
+before trusting a skip count.