Repository navigation
Validate pandas 3.1.1 and AnnData 0.13.4 together - #18
Draft
SecondBook5 wants to merge 20 commits into
Draft
SecondBook5 wants to merge 20 commits into
SecondBook5 wants to merge 20 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #17. #13 is closed. The velocity reference keeps NumPy 2.3.5. The candidate pairs pandas 3.1.1 with AnnData 0.13.4. It includes CellRank 2.3.3 for updated HDF5 registrations, LIANA 2.1.1 for pandas compatibility, and fixes to the UCell, CellOracle, MetaCell2 and manuscript-report handoffs.
Draft for scientific review. The initial paired comparison verified 19 of 20 candidate recipes. The extended pancreas recipe has since passed end to end from a clean wheel install, bringing the cumulative candidate coverage to 20 of 20. The other 19 results below come from the original comparison.
Scientific and figure changes
Reference runs
Each entry is a separate recipe on its documented input. The network-only SCC supplements shared a frozen upstream object. The main signaling and program versions were independently run from their documented SCC input through report generation. Blinded identity evaluation joined reference labels after freezing predictions. In the initial paired comparison, 325/340 matched PNGs have identical decoded pixels. Of 33 manuscript or PDF-only pages compared at 120 dpi, 28 pages match, three SCC communication pages change, and two extended-lineage pages were missing. The clean lineage run recovered both.
Installation and checks
The reference lineage environment uses the pinned upstream PyGPCCA revision already used in the CPU and GPU recipes, alongside the current Jinja2. CytoTRACE 2 runs the original 19-model ensemble in its own CPU worker with NumPy 1.26.4 and pandas 2.3.3. The analysis environment runs NumPy 2.3.5, pandas 3.1.1 and AnnData 0.13.4. Both environments pass
pip check. Clean installation instructionsAll ten PR CI jobs passed at 348a9a0, including the clean lineage install, the 412-file type check, strict documentation build and publication figure checks. The full suites on Python 3.12 and 3.13 each passed 4,175 tests, skipped 194 and reached 82.85% coverage. The container demo and security checks also passed. The installed wheel built from that commit ran the pancreas workflow in 3 min 21 s. Both restored PDF pages match baseline pixel for pixel at 150 dpi, and all nine diagnostic PNGs match exactly.
Patient-level SCC review
Of 973 sender–receiver–ligand–receptor combinations measured in at least five complete tumor/normal pairs, 801 retained their median direction and 172 reversed. Neither version yielded a 5% FDR discovery by paired sign test. The keratinocyte TGFB1 → fibroblast ENG direction persists in 5/5 patients. The Tensor factor 1 tumor decline falls from 6/8 to 4/8. CD8 T-cell HMGB1 → keratinocyte SDC1 gained three tumor-library top-100 calls but its paired support stays 2/4. The review includes named reversals, all donor differences, multiple-test results and an editable figure. Donor evidence and figure
GPU workstation validation
A fresh install of the GPU lock ran the installed wheel with pandas 3.1.1 and AnnData 0.13.4. A separately installed RAPIDS worker uses cuDF 26.10.2 with supported pandas 3.0.6. Both installs pass
pip check. The normal CLI completed six selected stages on 11,352 real human pancreas cells with CPU fallback disabled. The run records identify GPU PCA, Harmony, neighbors/Leiden and UMAP, and Lightning confirms CUDA scVI and scANVI training. Cross-study nearest-neighbor cell-type agreement matched the earlier reference for PCA (83.95%), scVI (81.24%) and label-informed scANVI (89.75%). The final lock ran from input to HTML report in 2 min 8 s on one RTX 4000 Ada. The full import check passed for all 34 analysis and five RAPIDS packages after declaring LIANA’s pooch dependency. A separate CPU LIANA run from the same installed GPU environment exactly reproduced all 54,450 magnitude and specificity scores across the 18 SCC libraries. The preliminary and final installs gave 18 and 17 Leiden clusters (adjusted Rand index 0.932). Repeating the final lock with the same seed gave 17 clusters again, adjusted Rand index 0.937, identical PCA/scVI/scANVI cross-study agreement, and a different UMAP rendering. Portable recipe, editable figure, per-cell data and verificationAll ten PR CI jobs passed at
e6c0ddb, alongside the container, security and lock checks. The final local regression suite passed 4,257 tests and skipped 113.CPU installation, restart behavior and consolidation
The general CPU lock runs pandas 3.1.1, AnnData 0.13.4 and scib-metrics 0.5.10. The installed pancreas workflow exports all 18 integration scores and links their CSV in its HTML report. On 11,352 cells, resuming after preprocessing reproduced the counts, normalized expression, PCA, Harmony, UMAP, Leiden assignments, all 18 scores and every pixel of nine PNGs. A changed score table was detected and rebuilt. Shuffling the cell-type labels forced a fresh measurement and changed Harmony NMI from 0.738 to 0.007. On the frozen SCC input, all 54,450 candidate magnitude and specificity ranks across 18 libraries were reproduced. CPU study configuration and results
Eleven methods share expression contracts. The lineage methods share their embedding selector, three figure families share the signed color scale, and annotation stages share donor and condition column resolution. The endocrine pancreas rerun reproduced all 3,696 potency scores, six fate probabilities per cell, five trajectory PNGs, and both publication pages. Lineage comparison
Stage consolidation and publication figures
All 29 stages that dispatch registered methods now declare their default method in one line. The shared stage handles method selection for planning and execution. Differential expression, differential abundance and enrichment resolve the primary study contrast through the same code. The two installed wheels returned identical method selections and applicable contrast settings across all 21 study configurations. Stage architecture
The before-and-after Kang runs completed all 18 selected stages on eight paired donors. All 24,562 cells retained their PCA and UMAP coordinates, Leiden assignments and cell types. All 12 checked differential-analysis and publication source tables matched byte for byte, including the six cell populations with positive interferon enrichment. Six manuscript PDFs matched when rendered. Label placement changed on the pooled volcano, dendritic-cell volcano and cell-type UMAP after switching to a fixed iteration limit and allowing clearance for crowded marker labels. Two fresh runs with the final wheel produced identical pixels in all 63 PNGs and six rendered publication PDFs.
The final local regression suite passed 4,261 tests with 113 skips. Ruff, mypy, installed CPU and GPU acceptance, and GitHub CI on Python 3.12 and 3.13 passed. PR checks
Remaining review
Review the previously reported SCC ranking reversals and revised Tensor interpretation before merging. Studies with checkpoints from fingerprint schema v5 or earlier need a fresh upstream run to produce compatible v6 checkpoints.