Skip to content

Validate pandas 3.1.1 and AnnData 0.13.4 together - #18

Draft
SecondBook5 wants to merge 20 commits into
mainfrom
codex/pandas-anndata-upgrade
Draft

SecondBook5 wants to merge 20 commits into
mainfrom
codex/pandas-anndata-upgrade

Conversation

@SecondBook5

@SecondBook5 SecondBook5 commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Replaces #17. #13 is closed. The velocity reference keeps NumPy 2.3.5. The candidate pairs pandas 3.1.1 with AnnData 0.13.4. It includes CellRank 2.3.3 for updated HDF5 registrations, LIANA 2.1.1 for pandas compatibility, and fixes to the UCell, CellOracle, MetaCell2 and manuscript-report handoffs.

Draft for scientific review. The initial paired comparison verified 19 of 20 candidate recipes. The extended pancreas recipe has since passed end to end from a clean wheel install, bringing the cumulative candidate coverage to 20 of 20. The other 19 results below come from the original comparison.

Scientific and figure changes

  • SCC communication. Full installed-wheel runs produced all four communication methods, both regulatory methods and seven verified manuscript panels. Both versions scored the same 54,450 interactions across 18 libraries. Changing only LIANA reproduced the magnitude rank shift (median Spearman ρ = 0.666, median 94/100 leading interactions retained). The matched AnnData and pandas controls preserved all ranks exactly.
  • Donor evidence. Both analyses favor TGFB1–ENG in tumor in all five matched patients. The first tensor factor declined in six of eight pairs on the baseline and four on the candidate. The MultiNicheNet products and NicheNet target weights agree. All three regulatory-panel sources agree to floating-point precision. Actual manuscript panels, donor measurements and editable ranking figure
  • SCC programs. All 9,916 metacell assignments agree. Both runs recovered 239 metacells, 338 outliers, five coexpression modules and six NMF programs. DIALOGUE MCP2 had ten donors and mean stability 0.161 over five resamples in both.
  • Kang and Hao. All 22,503 Kang edgeR estimates and all 8,729 adjusted-significance decisions agree. CHOIR/sc-SHC gave five B-cell subclusters in both baseline executions and four then six in candidate executions. All 161,764 Hao calls agree, including the 3,213 abstentions. CD8 recall was 21.0% and NK recall 99.5% in both.
  • Mouse endocrine lineage. All 3,696 cells received a CytoTRACE 2 score and all potency classes agree with baseline. The largest absolute score difference was 4.19 × 10⁻⁹ or less. Median potency fell from 0.300627 in ductal cells to 0.083378 in Ngn3-high progenitors and 0.048151 in beta cells. Both Palantir publication pages are present. Per-cell evidence, plotted data and figure pages
  • Other references. All 2,961 held-out pancreas transfer labels agree (85.01% accuracy). The four-donor cell/nucleus null result gives p = 0.875 in both. K562 GATA1 held-out rank correlation stays 0.125, with the same 25 significant network calls.

Reference runs

Each entry is a separate recipe on its documented input. The network-only SCC supplements shared a frozen upstream object. The main signaling and program versions were independently run from their documented SCC input through report generation. Blinded identity evaluation joined reference labels after freezing predictions. In the initial paired comparison, 325/340 matched PNGs have identical decoded pixels. Of 33 manuscript or PDF-only pages compared at 120 dpi, 28 pages match, three SCC communication pages change, and two extended-lineage pages were missing. The clean lineage run recovered both.

Study Candidate evaluation Identical / matched PNGs
PBMC droplets Verified 10/11
Matched cells and nuclei Verified 34/36
Kang paired response Verified 60/63
Kang expression panels Verified 10/10
Kang population review Verified 3/5
Kang sensitivity Verified 7/7
Kang blinded identity Identity evaluated 35/36
Hao blinded identity Identity evaluated 35/35
Human pancreas integration Verified 6/6
Human pancreas label transfer Verified 11/11
Human pancreas latent models Verified 6/6
Mouse pancreas lineage Verified 1/1
Mouse pancreas fates Verified 9/10
Mouse pancreas extended lineage Verified on the repaired clean install 9/9
Mouse pancreas CellOracle Verified 9/9
K562 GATA1 perturbation Verified 1/1
SCC communication and regulation Verified 18/20
SCC population network Verified 24/25
SCC gene-channel network Verified 24/25
SCC multicellular programs Verified 13/14

Installation and checks

The reference lineage environment uses the pinned upstream PyGPCCA revision already used in the CPU and GPU recipes, alongside the current Jinja2. CytoTRACE 2 runs the original 19-model ensemble in its own CPU worker with NumPy 1.26.4 and pandas 2.3.3. The analysis environment runs NumPy 2.3.5, pandas 3.1.1 and AnnData 0.13.4. Both environments pass pip check. Clean installation instructions

All ten PR CI jobs passed at 348a9a0, including the clean lineage install, the 412-file type check, strict documentation build and publication figure checks. The full suites on Python 3.12 and 3.13 each passed 4,175 tests, skipped 194 and reached 82.85% coverage. The container demo and security checks also passed. The installed wheel built from that commit ran the pancreas workflow in 3 min 21 s. Both restored PDF pages match baseline pixel for pixel at 150 dpi, and all nine diagnostic PNGs match exactly.

Patient-level SCC review

Of 973 sender–receiver–ligand–receptor combinations measured in at least five complete tumor/normal pairs, 801 retained their median direction and 172 reversed. Neither version yielded a 5% FDR discovery by paired sign test. The keratinocyte TGFB1 → fibroblast ENG direction persists in 5/5 patients. The Tensor factor 1 tumor decline falls from 6/8 to 4/8. CD8 T-cell HMGB1 → keratinocyte SDC1 gained three tumor-library top-100 calls but its paired support stays 2/4. The review includes named reversals, all donor differences, multiple-test results and an editable figure. Donor evidence and figure

GPU workstation validation

A fresh install of the GPU lock ran the installed wheel with pandas 3.1.1 and AnnData 0.13.4. A separately installed RAPIDS worker uses cuDF 26.10.2 with supported pandas 3.0.6. Both installs pass pip check. The normal CLI completed six selected stages on 11,352 real human pancreas cells with CPU fallback disabled. The run records identify GPU PCA, Harmony, neighbors/Leiden and UMAP, and Lightning confirms CUDA scVI and scANVI training. Cross-study nearest-neighbor cell-type agreement matched the earlier reference for PCA (83.95%), scVI (81.24%) and label-informed scANVI (89.75%). The final lock ran from input to HTML report in 2 min 8 s on one RTX 4000 Ada. The full import check passed for all 34 analysis and five RAPIDS packages after declaring LIANA’s pooch dependency. A separate CPU LIANA run from the same installed GPU environment exactly reproduced all 54,450 magnitude and specificity scores across the 18 SCC libraries. The preliminary and final installs gave 18 and 17 Leiden clusters (adjusted Rand index 0.932). Repeating the final lock with the same seed gave 17 clusters again, adjusted Rand index 0.937, identical PCA/scVI/scANVI cross-study agreement, and a different UMAP rendering. Portable recipe, editable figure, per-cell data and verification

All ten PR CI jobs passed at e6c0ddb, alongside the container, security and lock checks. The final local regression suite passed 4,257 tests and skipped 113.

CPU installation, restart behavior and consolidation

The general CPU lock runs pandas 3.1.1, AnnData 0.13.4 and scib-metrics 0.5.10. The installed pancreas workflow exports all 18 integration scores and links their CSV in its HTML report. On 11,352 cells, resuming after preprocessing reproduced the counts, normalized expression, PCA, Harmony, UMAP, Leiden assignments, all 18 scores and every pixel of nine PNGs. A changed score table was detected and rebuilt. Shuffling the cell-type labels forced a fresh measurement and changed Harmony NMI from 0.738 to 0.007. On the frozen SCC input, all 54,450 candidate magnitude and specificity ranks across 18 libraries were reproduced. CPU study configuration and results

Eleven methods share expression contracts. The lineage methods share their embedding selector, three figure families share the signed color scale, and annotation stages share donor and condition column resolution. The endocrine pancreas rerun reproduced all 3,696 potency scores, six fate probabilities per cell, five trajectory PNGs, and both publication pages. Lineage comparison

Stage consolidation and publication figures

All 29 stages that dispatch registered methods now declare their default method in one line. The shared stage handles method selection for planning and execution. Differential expression, differential abundance and enrichment resolve the primary study contrast through the same code. The two installed wheels returned identical method selections and applicable contrast settings across all 21 study configurations. Stage architecture

The before-and-after Kang runs completed all 18 selected stages on eight paired donors. All 24,562 cells retained their PCA and UMAP coordinates, Leiden assignments and cell types. All 12 checked differential-analysis and publication source tables matched byte for byte, including the six cell populations with positive interferon enrichment. Six manuscript PDFs matched when rendered. Label placement changed on the pooled volcano, dendritic-cell volcano and cell-type UMAP after switching to a fixed iteration limit and allowing clearance for crowded marker labels. Two fresh runs with the final wheel produced identical pixels in all 63 PNGs and six rendered publication PDFs.

The final local regression suite passed 4,261 tests with 113 skips. Ruff, mypy, installed CPU and GPU acceptance, and GitHub CI on Python 3.12 and 3.13 passed. PR checks

Remaining review

Review the previously reported SCC ranking reversals and revised Tensor interpretation before merging. Studies with checkpoints from fingerprint schema v5 or earlier need a fresh upstream run to produce compatible v6 checkpoints.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant