Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,3 +69,34 @@ Example user-tier `~/.config/cortexkit/synapse.jsonc` (project configs must omit

Tests can point at a file with `SYNAPSE_CONFIG_PATH`. Only one synapse module
per machine (singleton lease); a second instance refuses to start.

### Owned-CUDA hardware floor

`ck-synapse-worker-cuda` implements `--probe-floor` (hidden, like the
`--test-abort*` surfaces). It prints one JSON object and exits 0:

```json
{"driver_api": 13030, "compute_capability": {"major": 8, "minor": 9}}
```

The module probes the configured `worker_bin`, the engine's worker-binary
environment override, or the sibling `ck-synapse-worker-cuda`, in that order.
It caches one result per process unless both environment readings parse
successfully. The child wait is bounded to 10 seconds; stdout is capped at
4096 bytes and each pipe completion wait is bounded to another 100 ms.
A missing binary, non-zero exit, timeout, or invalid output produces
`HardwareUnavailable`. Refusal and model evidence carry diagnostic context,
including the last 4096 bytes of stderr when available, under `observed`.
Failed probes do not fabricate numeric hardware readings.

The environment overrides the probe only as a complete, parseable pair.
Otherwise both readings come from the probe; partial overrides are not merged:

- `SYNAPSE_CUDA_DRIVER_API` (alias `CUDA_DRIVER_API`) — the raw CUDA **driver
API** integer from `cuDriverGetVersion()`, not the marketing driver version.
For example, a measured driver API value is `13030`. `610.88` is not a valid
API integer; without a parseable alias, it causes fallback to the probe.
- `SYNAPSE_CUDA_COMPUTE_CAPABILITY` (alias `CUDA_COMPUTE_CAPABILITY`) — device
0's compute capability as `major.minor`, for example `8.9`.
- `SYNAPSE_CUDA_PACKAGING_DRIVER` — optional; the driver string a packaging
build was tested against, carried into the refusal for diagnostics.
61 changes: 60 additions & 1 deletion crates/synapse-engine-cuda/src/cuda.rs
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,51 @@ mod enabled {
Ok(())
}

/// Read the driver API version and device 0's compute capability.
///
/// This runs before an owned-CUDA load is admitted, so it deliberately
/// touches nothing else: no context is retained, no model is loaded, and
/// no weights are mapped. The reading is what the floor predicate is
/// applied to, which is why it reports the raw numbers rather than a
/// verdict.
pub fn probe_hardware_floor() -> Result<crate::HardwareFloorProbe> {
cuda_driver_check(unsafe { cuInit(0) }, "cuInit")?;
let mut driver_api = 0;
cuda_driver_check(
unsafe { cuDriverGetVersion(&mut driver_api) },
"cuDriverGetVersion",
)?;
let mut device = 0;
cuda_driver_check(unsafe { cuDeviceGet(&mut device, 0) }, "cuDeviceGet")?;
let mut major = 0;
cuda_driver_check(
unsafe {
cuDeviceGetAttribute(
&mut major,
CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR,
device,
)
},
"cuDeviceGetAttribute(COMPUTE_CAPABILITY_MAJOR)",
)?;
let mut minor = 0;
cuda_driver_check(
unsafe {
cuDeviceGetAttribute(
&mut minor,
CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR,
device,
)
},
"cuDeviceGetAttribute(COMPUTE_CAPABILITY_MINOR)",
)?;
Ok(crate::HardwareFloorProbe {
driver_api: driver_api as u32,
compute_major: major as u32,
compute_minor: minor as u32,
})
}

pub struct MiniLmContext {
binding: DeviceBinding,
raw: NonNull<c_void>,
Expand Down Expand Up @@ -465,8 +510,16 @@ mod enabled {
}
}

/// `CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR` from `cuda.h`.
const CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR: i32 = 75;
/// `CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR` from `cuda.h`.
const CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR: i32 = 76;

unsafe extern "C" {
fn cuInit(flags: u32) -> i32;
fn cuDriverGetVersion(version: *mut i32) -> i32;
fn cuDeviceGet(device: *mut i32, ordinal: i32) -> i32;
fn cuDeviceGetAttribute(value: *mut i32, attrib: i32, device: i32) -> i32;
fn cuCtxGetDevice(device: *mut i32) -> i32;
fn cuCtxSetCurrent(context: *mut c_void) -> i32;
fn cuDevicePrimaryCtxRetain(context: *mut *mut c_void, device: i32) -> i32;
Expand Down Expand Up @@ -551,6 +604,10 @@ mod enabled {
bail!("owned CUDA requires a non-macOS build with cargo feature `cuda`")
}

pub fn probe_hardware_floor() -> Result<crate::HardwareFloorProbe> {
bail!("owned CUDA requires a non-macOS build with cargo feature `cuda`")
}

pub struct MiniLmContext;
impl MiniLmContext {
pub fn new(_graphs: bool) -> Result<Self> {
Expand Down Expand Up @@ -629,4 +686,6 @@ mod enabled {
}
}

pub use enabled::{ensure_available, MiniLmContext, ModernBertContext, Qwen3Context};
pub use enabled::{
ensure_available, probe_hardware_floor, MiniLmContext, ModernBertContext, Qwen3Context,
};
13 changes: 13 additions & 0 deletions crates/synapse-engine-cuda/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ use synapse_core::{
mod cuda;
mod model;

pub use cuda::probe_hardware_floor;

pub const ENGINE_VERSION: &str = "owned-cuda-v1";
/// The source revision from which the CUDA kernels were ported.
pub const KERNEL_REVISION: &str = "4d0ded67c30286fe2be37cc7413359ad745dd751";
Expand Down Expand Up @@ -183,6 +185,17 @@ pub fn build_identity(family: ModelFamily, dtype: StorageDType) -> CudaBuildIden
}
}

/// A hardware-floor reading taken before any owned-CUDA worker is spawned.
///
/// Carried separately from [`device_meets_floor`] so the caller can log or
/// refuse on the observed values rather than on a bare boolean.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub struct HardwareFloorProbe {
pub driver_api: u32,
pub compute_major: u32,
pub compute_minor: u32,
}

/// Hardware-floor predicate used by capability probes before worker creation.
#[must_use]
pub fn device_meets_floor(driver_api: u32, compute_major: u32, compute_minor: u32) -> bool {
Expand Down
Loading
Loading