Fix single-cell counting semantics and harden numerical parity validation

[Jaureguy760]

Single-cell allele counts could wrap above 65,535, automatic sample selection skipped heterozygote filtering, and shared-SNP differential analysis incorrectly applied a region minimum to individual SNPs. This PR fixes those behaviors and makes numerical compatibility claims distinguish original narrow-count input from explicitly widened references.

Changes

• Store sparse counts/totals as UInt64 and construct pandas metadata without the undeclared PyArrow dependency.
• Normalize repeated/comma/file sample arguments; apply identical genotype filtering and feature reindexing for automatic and explicit sample selection.
• Restore archived shared-SNP eligibility and sequential unphased likelihood arithmetic, with a stable fallback when probability products underflow.
• Compare validation count columns exactly before float conversion, including one-count differences above 2**53.
• Keep the archive loader unchanged by default. Independent opt-in adapters widen only single-cell sum keywords or the three bulk CSV count dtypes. Per-row diagnostics retain strict legacy failures and distinguish count, likelihood, estimate, p-value and multiple-testing differences. Complementary estimates are candidates until independently checked against the actual objective.
• Add 70-digit Decimal beta-binomial checks independent of SciPy special functions, scalar/array and balanced-null checks, wide raw-count/pseudocounted-depth checks, and symmetric/asymmetric unphased-anchor regressions.
• Correct backend and AnnData documentation: Rust handles counting and bulk inference; single-cell inference remains Python/SciPy. Both use BH, with separate implementations.

Validation

Final-head Linux Python 3.11: 711 passed, 76 skipped, 1 deselected. All seven final-head workflows pass: CI (Python 3.10–3.13, Rust tests, Clippy and formatting), Documentation, CodeQL, Security, Build (Linux/macOS wheels), Ecosystem (Galaxy/Nextflow), and Container CI (native amd64/arm64). Local numerical-kernel suite: 39 passed. Local Rust: 132 passed, 7 ignored, plus 3 ignored doctests. Ruff lint/format, release metadata and diff checks pass.

Committed synthetic coverage includes 24 independent three-way counting comparisons, eight BAM/VCF/barcode/peak -> counting CLI -> H5AD -> analysis CLI cases, and 144 expanded single-cell statistical configurations. Frozen expectations are unchanged.

Real bulk

The existing 32-configuration matrix supplies wide-integer DataFrames to archived Python; it never established strict UInt16 CSV parity. Tests now additionally execute original and widened CSV readers, including both models with explicitly documented linear shape/phase reference repairs.

All 16 widened CSV configurations pass. The untouched-width reference fails strict comparison in all 16; all 485 failing case-row comparisons are retained and localized to the reference-width intervention, with zero unexplained widened-reference failures. All three complementary-mu candidates are independently confirmed symmetric at the retained balanced anchor. These are repeated configuration-row comparisons, not 485 unique loci.

Real single-cell

Public input: 10x GM12878/EL4 ATAC v2 Chromium Controller, Cell Ranger ATAC 2.1.0, GRCh38, paired with HG001 GIAB v3.3.1 phased genotypes. Chromosome 21: 8,760,471 alignment records; 4,804 human singlets; 27,182 heterozygous biallelic PASS SNPs; 1,554 source peaks; 1,190 peak-overlapping SNPs, 1,185 phased. Alignment records are preserved; genotype contig names alone are translated.

Archived/current Python and Rust counts match exactly at threads 1/4, in SNP-only and peak-restricted modes. The public counting CLI prohibits Python fallback during validation. H5AD matrices, SNP/barcode identities/order and peak memberships match.

All 54 widened-reference statistical configurations match with zero reported numerical deltas. The unmodified archive passes 20/54; the other 34 remain strict failures. All 669 failing case-row comparisons have width-intervention classifications; all 100 complementary-estimate candidates are independently certified using the actual individual or joint anchor symmetry and separate log-gamma objective evaluation. Dispersion-fit differences are recorded separately. No tested q<0.05/q<0.10 decisions change.

All 24 public analysis CLI runs and six genotype/outlier preprocessing comparisons pass. Groups of 2,445/2,359 cells use SHA-256(barcode) parity and are technical partitions, not biological conditions.

Numerical interpretation and limits

UInt16 arrays trigger float32 floor/log intermediates in the pinned SciPy beta-binomial floor/log/betaln expression, while scalar and wide-count paths differ. A balanced 7/14 observation can therefore receive unequal null/alternative likelihoods in the original archive. Independent high-precision calculations support consistent float64 evaluation. The observed real single-cell maximum SNP depth is only 1,875: floating-point dispatch, not count overflow, explains this dataset's differences.

No likelihood definition, chi-square df=1, BH procedure, pseudocount policy or dispersion scope is changed by the validation follow-up. No output flipping, dataset-specific production branches or tolerance relaxation is introduced. Narrow-reference failures are not relabeled as parity passes.

Validation is limited to one chromosome and donor genotype on provider-aligned BAMs; it does not certify whole-genome behavior, a new WASP remapping pass, biological calibration/power or Rust skip-diagnostic metadata. Full historical counting CLI execution is not claimed: its original counter and intersection parser run independently, while bcftools is absent locally.

Data, input hashes, scripts and detailed classifications remain outside Git. The only new repository file is the original generated-input counting regression test. Native Rust single-cell inference and Rust UInt32 overflow hardening remain separate work. PR #139 remains separate; neither PR is merged and nothing is published.

Final remote head: 1fe407a; source tree: e10a44ef6eb9bb45bc309d596288fd706f992125. Local and remote trees match. Real-data reruns used the preceding validation commit; the final commit adds only two tests, with identical runtime source.

[Jaureguy760]

Superseded by #140, now merged into dev at 4cd820b9b141e61dd8fda7fde32026b87969e75d. The replacement includes these changes, the combined integration fixes and final-head validation across all seven PR workflows. Closing this PR as superseded; no release has been tagged or published.