Repository navigation
Add donor-local dispersion while preserving Python imbalance tests - #133
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rust bulk counting and inference differed from archived Python in SNP assignment, pseudocount reporting, phase decisions and numerical likelihood evaluation. This PR restores that behavior and adds donor-local inference with either shared/global or per-donor dispersion.
Behavior
Validation
Final head: c944c04; tested tree: 8dfd91aac1f3d512248d215d2b47674bc8f65a5a. Local and remote trees match exactly.
All 32 real ATAC comparisons pass STRICTLY on macOS arm64 and Linux x86_64, including raw mu, exact counts, likelihoods, dispersion predictions, LRT p-values, BH and significance checks. Tolerances remain rtol=atol=5e-5. Sixteen additional conditional cases execute native archived Python with identical supplied nuisance parameters; the original frozen Mac expectations remain unchanged and are also checked in their pinned environment. No complementary-output exemption is used for strict acceptance.
Data: GM12878 ATAC ENCSR637XSC, BAM ENCFF962FMH, peaks ENCFF452ZBX, matched HG001 GIAB NISTv3.3.1 GRCh38 genotypes. Source MD5 checks verified. Counting matches executed archived Python on 27,182 SNPs and 556 peak rows at threads 1 and 4. The BAM/VCF/peaks counting CLI reproduced all 556 rows and memberships; eight real peak CLI comparisons pass. Compressed measured inputs and source/table checksums are committed.
Local: 474 Python tests passed, 82 skipped, 1 deselected; 132 Rust tests passed, 7 ignored. Final Linux Python 3.11: 480 passed, 76 skipped, 1 deselected. Final core CI is green on Python 3.10–3.13, Rust tests, current-stable Clippy, lint and version checks. The 49 genotype/boundary regressions include 16 normalized-GT duplicate cases. Heterogeneous donor comparisons independently fit both models with overlaps, phase modes and pseudocounts. Fresh committed wheel rerun passes all 32 cases and includes both third-party notices.
All final-head workflows are green: core CI, Build, Container CI, Ecosystem, Documentation, CodeQL and Security. Native linux/amd64 and linux/arm64 container builds and all ATAC smoke/label checks pass. Container smoke tests cover reordered ATAC headers, grouping, phasing, raw counts and Python/Rust version agreement; image builds carry checked package-version and source-revision labels. Both native architectures passed these checks on the final head.
Limits and integration
One real donor/chromosome plus controlled heterogeneous donors; not biological calibration or a new WASP remapping pass. Archived linear shape repairs and phase forwarding are explicit. Phased real-data comparisons select truly phased SNPs; the one mixed 0/1 peak SNP is retained for unphased tests and rejected by both implementations when phased inference is requested.
Python itself selects different raw directions for some symmetric unphased cases across numerical environments. Rust matches the executed reference on each validated platform; universal bitwise/cross-platform direction agreement is not promised. Endpoint regularization and underflow fallback extend the archive's numerical failures.
Review/integrate #135 first, then refresh this PR against dev and rerun required checks. A local integration simulation has no conflicts and retains the exact tested tree. No merge or release is included; published packages/images remain unchanged. The release/Singularity dry run remains a later release gate. Full report: docs/BULK_PARITY_VALIDATION.md.