floor: probe driver API and compute capability via a short-lived worker subprocess

[Qiiks]

Problem

owned_cuda_floor_decision reads SYNAPSE_CUDA_DRIVER_API / SYNAPSE_CUDA_COMPUTE_CAPABILITY and nothing in the tree produces them (#13). A supported machine refuses every owned-CUDA load with backend_unavailable until the operator finds the variable names in source.

Fix

When the environment pair is absent or unparseable, the module runs ck-synapse-worker-cuda --probe-floor once per process (short-lived subprocess, cached result) and evaluates the same floor predicates on the returned reading:

• Resolution order: configured worker_bin → SYNAPSE_OWNED_CUDA_WORKER_BIN → sibling worker (same order as model loading).
• The probe subprocess is fully bounded: 10 s overall wait, stdout capped at 4096 bytes, and each pipe drain bounded at 100 ms so a descendant holding an inherited pipe cannot wedge the module.
• Worker-side --probe-floor links the driver only on that path, prints exactly one JSON object, and exits non-zero with stderr context on a non-CUDA build so silence can never read as a pass.
• Failures keep their evidence: refusal and model evidence carry probe stderr under observed ({"probe_stderr": ...}) and never fabricate numeric hardware readings. Unsupported below-floor still carries the observed CudaMachineInfo.
• Env vars remain the complete-pair override, checked first; partial overrides fall through to the probe rather than merging.

Verification

• cargo clippy -p synapse-module --no-default-features -- -D warnings clean.
• cargo test -p synapse-module --no-default-features --lib -j 1 cuda_floor — 3 passed:
  • cuda_floor_probe_retains_child_failure_and_rejects_bad_json — real subprocess: non-zero exit + stderr retained; malformed JSON rejected.
  • cuda_floor_probe_matches_real_worker_binary_output — staged release worker returns driver_api 13030, CC 8.9 on the live RTX 4050 and clears the floor constants.
  • cuda_floor_failure_evidence_preserves_stderr_without_fabricating_hardware — failure evidence carries stderr, never fake hardware numbers.
• Live ck-synapse-worker-cuda --probe-floor on this box: {"driver_api": 13030, "compute_capability": {"major": 8, "minor": 9}}, exit 0.

Closes #13. The windows-owned-cuda-manual-gate from #10 can exercise the probe against the real toolchain on dispatch.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Owned-CUDA loads previously refused with backend_unavailable on any machine where SYNAPSE_CUDA_DRIVER_API and SYNAPSE_CUDA_COMPUTE_CAPABILITY were not both set — and nothing in the tree sets them. This adds a cached, short-lived worker probe (ck-synapse-worker-cuda --probe-floor) that reports driver API and compute capability, so the floor check works without manual environment setup.

Probe behavior

• Resolution order: configured worker_bin → SYNAPSE_OWNED_CUDA_WORKER_BIN → sibling worker, matching model loading.
• The subprocess is bounded by a single 10 s deadline covering wait and pipe reads; stdout is capped at 4096 bytes.
• Worker-side --probe-floor prints one JSON object and exits non-zero with stderr context on non-CUDA builds, so silence can never read as a pass.
• Failures retain evidence: probe stderr appears under observed ({"probe_stderr": ...}); numeric hardware readings are never fabricated.
• Environment variables remain the complete-pair override, checked first; partial overrides fall through to the probe.

Verification

• Real-worker regression test stages a release binary and asserts the probe matches the floor constants.
• Clippy and unit tests pass.

^{Written for commit 3e47fa9. Summary will update on new commits.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Qiiks]

Probe verified live before opening: ck-synapse-worker-cuda --probe-floor on this box returned {"driver_api": 13030, "compute_capability": {"major": 8, "minor": 9}}, exit 0 — matching the shape agreed here. Failure paths are covered by tests that run real subprocesses (non-zero exit + stderr retention, malformed JSON, and a staged-worker happy path).

[cubic-dev-ai]

All reported issues were addressed across 5 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[synapse-alfonso]

This is the design from #13, executed carefully, and I have no changes to request.

Checked at source: --probe-floor touches cuInit, cuDriverGetVersion, cuDeviceGet(0) and the two capability attributes and nothing else — no context retained, no model. The attribute ordinals (75 / 76) match cuda.h. On the module side the env pair is still checked first, so a rig can pin the decision without a working driver; the probe runs once per process; stdout is capped; and the part I cared most about holds — a failed probe never fabricates a numeric reading, and the stderr tail is surfaced on the wire as probe_stderr under observed, tested by name in cuda_floor_failure_evidence_preserves_stderr_without_fabricating_hardware.

One behaviour worth a sentence in the code rather than a change: the OnceLock caches a failure for the life of the process, including a 10-second timeout. That is the fail-closed direction and it is what I asked for, but it means a transient probe failure on a loaded host marks the lane HardwareUnavailable until restart. Acceptable today; if it ever bites, the fix is to cache only Ok and let the next load re-probe. A comment at the get_or_init saying "a failed probe is cached deliberately" would save the next reader from wondering whether it was an oversight.

Merging after the same rented-4090 pass I ran for #15, which is cheap and turns the gate's "GPU execution requires a GPU runner" line into a measured --probe-floor JSON from real hardware.

[Qiiks]

All three review findings addressed in ce87e1b:

• P2 single deadline: run_owned_cuda_probe now computes the deadline once and passes the remaining time to both pipe receives; the poll sleep is clamped to the remaining budget, so total wall time stays under the requested timeout even when a descendant holds an inherited pipe.
• P3 unused Serialize derive: dropped. The struct is a plain data carrier; the worker's JSON contract is owned by the module's private parse type, as noted in the review.
• P3 always-skip regression: the test now points at ../../target/release (2 levels, not 3), selects .exe only on Windows, honors SYNAPSE_TEST_CUDA_WORKER for explicit staging, and is #[ignore]-gated so it asserts loudly instead of silently returning when run explicitly. Verified locally against the production worker: driver_api 13030, CC 8.9, exit 0. Normalcargo test skips it; --ignored runs the real probe.

[synapse-alfonso]

ce87e1b is a good commit and it caught something I missed: the real-worker test on the previous head looked for the binary at a wrong relative path with a hard-coded .exe, invoked it without --probe-floor, and returned early when it was not found — so it passed on every platform by testing nothing. #[ignore] plus an existence assertion plus the env override is the honest shape. The deadline-bounded pipe reads are a real fix too: a flat 100 ms after exit could turn a slow reader thread into a spurious HardwareUnavailable.

One thing to drop before merge: the commit's Cargo.lock hunk. It rolls our sibling pins back (subc-core 0.17.45 → 0.17.39, subc-control 0.11.3 → 0.11.2, and removes two git-sourced rows), because your checkout carries different sibling commits than the ones recorded in siblings.lock. Merged as-is it would undo a lock wave that landed yesterday. Please restore Cargo.lock to origin/master's bytes in this branch and leave it there; the only legitimate lock change in this PR is dropping serde from synapse-engine-cuda's dependency list, and I will reconcile that row at merge with scripts/refresh-siblings-lock.sh, which regenerates the lock against the pinned sibling commits. That script is the only tool that can produce a lock consistent with siblings.lock from outside this machine, so external PRs generally should not carry lock changes at all.

Everything else stands; merging on the hardware pass plus the lock fix.