Repository navigation
gpuMi, gpuRowTtest, gpuCorPerm, Spearman ranks: the method fixes from widestats - #53
Merged
Merged
Conversation
… widestats gpuMi gave NaN (undefined behaviour, in fact: floor(NaN) cast to int) for a constant variable, built its B-spline weights in the requested precision from narrowed data (a value on a knot, as discrete data put them, could land in the neighbouring bin in single), accepted splineOrder > bins, subtracted entropies, and looped over samples per pair. Now: weights in double from the double data, every pair's joint distribution from one GEMM per tile, the information in log form from each block's own marginals, exactly 0 for a constant variable, never below 0, 1 <= splineOrder <= bins <= 64 checked, defaults Daub et al.'s bins = 10, splineOrder = 3. Help page rewritten: the data need not lie in [0, 1] (the kernel always rescaled), order is not degree. gpuRowTtest with equal groups compared a labelling's swap, computed by different roundings, against the observed t from the first block only; for strongly separated rows the swap could fail the allowance and the p-value came out too small. Labellings are canonical (sample 1 in group 1) and the observed t is recomputed with every block; a within-group sum of squares within 16 eps of its total is zero; a complete separation is +/-Inf with p 0 (Welch's df NA; Student's n - 2) instead of NaN; rows with no test stay out of the maxT maxima. gpuCorPerm's allowance is relative, as gpuRowTtest's. Spearman ranks are taken in double before any narrowing (gpuCor, gpuCorPairs, gpuCorPerm). Reviewed against widestats (weights to 1e-14, Daub's worked example, tiling at several tile sizes, the block loop for every block size); the full suite passes on an A5500. Tests pin the equal-group count and the complete-separation count exactly from the draws, constant variables, knot-valued data in single vs double, and the argument checks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second and last of the backports from widestats (after #52).
gpuMi. A constant variable gaveNaN(viafloor(NaN)cast to int, undefined behaviour); the B-spline weights were built in the requested precision from narrowed data, so in single a value sitting on a knot, as discrete data do, could land in the neighbouring bin;splineOrder > binswas accepted; entropies were subtracted; and every pair's joint distribution came from a per-pair loop over samples. Now the weights are computed in double from the double data (miWeights), every pair's joint distribution comes from one GEMM per tile of the weight matrices, the information is computed in log form from each block's own row and column sums (never below 0), a constant variable is exactly 0,1 ≤ splineOrder ≤ bins ≤ 64is checked, and the defaults are Daub et al.'sbins = 10, splineOrder = 3rather than plain binning into two bins. The help page no longer says the data must lie in [0, 1] (the kernel always rescaled by min and max) or calls the order a degree.gpuRowTtest. With equal groups, a labelling and its swap are the same test but were computed by different roundings and compared against the observed t from the first block only; for strongly separated rows the swap could miss the allowance and the p-value came out too small. Labellings are now canonical (sample 1 in group 1) and the observed t is recomputed with every block of permutations, so an equivalent labelling always counts; a within-group sum of squares within 16 ε of its total is zero; a row constant within each group with the means apart is±Infwith p = 0 (Welch's dfNA, Student's n − 2) instead ofNaN; rows with no test stay out of the maxT maxima.Also.
gpuCorPerm's rounding allowance is relative, asgpuRowTtest's; Spearman ranks (gpuCor,gpuCorPairs,gpuCorPerm) are taken in double before any narrowing, so distinct values no longer tie in single.Verification. Full testthat suite on an A5500 (CUDA 12.6). Review against widestats with NumPy transcriptions: weights agree to 1e-14 across bins 5–64 and orders 1–4, Daub's worked example reproduced, tiling verified for tile 1, 2, 5, the permutation block loop simulated for every block size. New tests pin the equal-group count and the complete-separation count exactly from the RNG draws (the single-precision case is the one that failed before), constant variables, knot-valued data in single vs double, and the argument checks.
🤖 Generated with Claude Code