Skip to content

Add donor-local dispersion while preserving Python imbalance tests - #133

Merged
Jaureguy760 merged 25 commits into
devfrom
fix/donor-local-inference-20260721
Oct 15, 2026
Merged

Jaureguy760 merged 25 commits into
devfrom
fix/donor-local-inference-20260721

Conversation

@Jaureguy760

@Jaureguy760 Jaureguy760 commented Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator

Rust bulk counting and inference differed from archived Python in SNP assignment, pseudocount reporting, phase decisions and numerical likelihood evaluation. This PR restores that behavior and adds donor-local inference with either shared/global or per-donor dispersion.

Behavior

  • Preserve input-order SNP priority, read-name consumption before base lookup, overlapping feature observations, first-SNP unphased anchoring and phase-aware deduplication.
  • Apply raw-GT phasing checks after depth filtering, then deduplicate using the parsed first allele. When raw GT tokens collapse to one key, retain the original input for Rust's pre-dedup phase check while fitting, QC and output validation use the retained observations. Malformed mixed phased input raises an error.
  • Keep pseudocount default 1, honor explicit 0, report counts after per-observation pseudocount removal, and retain chi-square LRT with one degree of freedom and donor-local BH. Exact-SNV alternative p-values are rejected.
  • Use archived bounded Brent defaults and single-start Nelder-Mead. Linear fits preserve adjusted-depth coefficients through depth_center=-2*pseudocount and depth_scale=1; existing supplied four-field parameter sets keep their original meaning.
  • Use SciPy 1.13.1-compatible Cephes arithmetic and NumPy 1.26.4 reductions. Measured compiler contraction differs between Linux x86_64 and macOS arm64; Rust follows each reference policy. Valid interior rho is preserved. Exact endpoints and probability-product underflow retain documented finite extensions.
  • Pool only nuisance fitting for global dispersion. Effects, phasing decisions, result validation, provenance and BH stay donor-local. Includes Restore archived Python bulk SNP counting and analysis compatibility #135's counting, raw-count and single-cell compatibility changes.

Validation

Final head: c944c04; tested tree: 8dfd91aac1f3d512248d215d2b47674bc8f65a5a. Local and remote trees match exactly.

All 32 real ATAC comparisons pass STRICTLY on macOS arm64 and Linux x86_64, including raw mu, exact counts, likelihoods, dispersion predictions, LRT p-values, BH and significance checks. Tolerances remain rtol=atol=5e-5. Sixteen additional conditional cases execute native archived Python with identical supplied nuisance parameters; the original frozen Mac expectations remain unchanged and are also checked in their pinned environment. No complementary-output exemption is used for strict acceptance.

Data: GM12878 ATAC ENCSR637XSC, BAM ENCFF962FMH, peaks ENCFF452ZBX, matched HG001 GIAB NISTv3.3.1 GRCh38 genotypes. Source MD5 checks verified. Counting matches executed archived Python on 27,182 SNPs and 556 peak rows at threads 1 and 4. The BAM/VCF/peaks counting CLI reproduced all 556 rows and memberships; eight real peak CLI comparisons pass. Compressed measured inputs and source/table checksums are committed.

Local: 474 Python tests passed, 82 skipped, 1 deselected; 132 Rust tests passed, 7 ignored. Final Linux Python 3.11: 480 passed, 76 skipped, 1 deselected. Final core CI is green on Python 3.10–3.13, Rust tests, current-stable Clippy, lint and version checks. The 49 genotype/boundary regressions include 16 normalized-GT duplicate cases. Heterogeneous donor comparisons independently fit both models with overlaps, phase modes and pseudocounts. Fresh committed wheel rerun passes all 32 cases and includes both third-party notices.

All final-head workflows are green: core CI, Build, Container CI, Ecosystem, Documentation, CodeQL and Security. Native linux/amd64 and linux/arm64 container builds and all ATAC smoke/label checks pass. Container smoke tests cover reordered ATAC headers, grouping, phasing, raw counts and Python/Rust version agreement; image builds carry checked package-version and source-revision labels. Both native architectures passed these checks on the final head.

Limits and integration

One real donor/chromosome plus controlled heterogeneous donors; not biological calibration or a new WASP remapping pass. Archived linear shape repairs and phase forwarding are explicit. Phased real-data comparisons select truly phased SNPs; the one mixed 0/1 peak SNP is retained for unphased tests and rejected by both implementations when phased inference is requested.

Python itself selects different raw directions for some symmetric unphased cases across numerical environments. Rust matches the executed reference on each validated platform; universal bitwise/cross-platform direction agreement is not promised. Endpoint regularization and underflow fallback extend the archive's numerical failures.

Review/integrate #135 first, then refresh this PR against dev and rerun required checks. A local integration simulation has no conflicts and retains the exact tested tree. No merge or release is included; published packages/images remain unchanged. The release/Singularity dry run remains a later release gate. Full report: docs/BULK_PARITY_VALIDATION.md.

@Jaureguy760 Jaureguy760 changed the title feat: add donor-local inference and finalize 1.5.0 Add donor-local dispersion while preserving Python imbalance tests Oct 13, 2026
@Jaureguy760
Jaureguy760 merged commit e6ffea9 into dev Oct 15, 2026
28 checks passed
@Jaureguy760
Jaureguy760 deleted the fix/donor-local-inference-20260721 branch October 15, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant