Skip to content

Check that every kernel has a parity case - #71

Open
CanReader wants to merge 3 commits into
masterfrom
test/kernel-coverage
Open

CanReader wants to merge 3 commits into
masterfrom
test/kernel-coverage

Conversation

@CanReader

Copy link
Copy Markdown
Owner
  • remove the transpose kernels nothing calls
  • run cublas in full fp32 instead of tf32
  • check that every kernel has a parity case

What this changes

tests/cuda_parity.rs gains a table of every kernel symbol in ffi.rs with the test that covers it, and a test that runs without a GPU and fails when a kernel has no row, a row names a kernel that no longer exists, or a row names a test that is not in the file. Filling the table in showed which kernels only ran indirectly inside the model test, so elementwise, reductions and gemm_batched get direct cases for sub, div, the scalar ops, neg, abs, sqrt, log, powf, max, min, the batched tn layout and batched backward.

Two kernels turned out to have no caller at all: transpose (the op is a permute) and transpose_batched. Both are removed with their header lines, stubs, ffi declarations and wrappers.

The cuBLAS handle now uses CUBLAS_DEFAULT_MATH instead of CUBLAS_TF32_TENSOR_OP_MATH.

Why

The README says every CUDA kernel is tested against the CPU path. There were 57 symbols and 16 tests, and nothing checked that the two matched. Now something does.

The first direct test of the batched tn GEMM failed by 2.8e-4 against a 2e-4 tolerance. Against an f64 reference the CPU path is within 1.5e-7 and so is the GPU plain layout, but the GPU tn layout was off by up to 5.5e-4 on 70 of 192 elements: cuBLAS was picking a TF32 tensor-core kernel for that shape and layout, which the handle allowed. The other parity tests passed only because cuBLAS did not pick TF32 for their shapes. TF32 is three significant digits, which is not "the last few digits" the docs promise. On an RTX 4050 it is worth 1.2x to 1.6x on large square GEMMs (2.78 vs 4.43 ms at 2048, 29 vs 36 ms at 4096). Full FP32 is the default that matches the contract; TF32 can come back as an opt-in if wanted.

How it was verified

  • cargo test passes
  • cargo clippy --all-targets -- -D warnings is clean
  • Ran cuda_parity on a GPU
  • Trained an example in --release
  • No GPU here, CUDA side not runtime tested

All 17 parity cases pass on an RTX 4050 under full FP32, the new coverage test passes on the CPU build, cargo doc with -D warnings is clean, and each of the three commits is green on its own.

Checklist

  • Every new or changed backward rule has a gradcheck case
  • Every new kernel has a cuda_parity case
  • tensor/ops and autograd/ops still mirror each other
  • Public items have /// docs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant