Start with a library sample sheet, Cell Ranger outputs, a 10x H5 or Matrix Market matrix, or a count-bearing AnnData (.h5ad) file. A YAML file
identifies your donors, conditions, and requested analyses. A run writes an annotated object,
figures, result tables, and an HTML report. For comparisons between conditions, the study
design specifies the biological replicate and whether samples are paired.
CellQuorum has 41 stages and 82 selectable methods. The study design and available data determine which analyses can run.
The paired carcinoma walkthrough runs four communication methods and two regulatory methods together and produces donor-linked manuscript figures. The reference-study coverage matrix identifies the evidence for each stage and method. The public-study validation report shows which results held up, where they disagreed with the study and which biological claims remain unresolved.
The paired PBMC walkthrough takes public count data from eight donors through marker-based annotation, cell-state scoring, donor-aware expression, abundance and pathway analysis. One run produces per-cell-type volcano plots, a cross-cell-type expression heatmap, a Hallmark pathway heatmap, the underlying statistics and a linked report. After downloading the two inputs and installing the dependencies in the walkthrough, run:
cellquorum run -c configs/kang_paired_analysis.yamlTry the CPU workflow on synthetic data in a fresh Python 3.12 environment:
git clone https://github.com/SecondBook5/cellquorum.git && cd cellquorum
python3.12 -m venv .venv
source .venv/bin/activate # Linux, macOS, or WSL
python -m pip install -e .
cellquorum demo my_demo # a tiny synthetic dataset + config
cellquorum plan --config my_demo/config.yaml # what will run, and on which backends
cellquorum run --config my_demo/config.yaml -o my_demo/runOpen my_demo/run/reports/report.html. Expand an analysis under Analysis outputs to see its
files. The demo exercises QC, normalization, PCA, clustering, population summaries, and UMAP.
Start from a library sample sheet, a supported 10x count matrix, or an AnnData file with raw counts in X or a named layer.
cellquorum init my_study --input /path/to/study.h5ad
cellquorum plan -c my_study/config.yaml
cellquorum run -c my_study/config.yamlChoose discovery to consider every question the study supports, or questions to select analyses such as communication, regulation or lineage. The setup uses your metadata columns to identify biological replicates, conditions and pairing, and offers sensitivity analysis where applicable. The saved YAML can be shared and rerun. The plan shows method prerequisites before computation.
A single filtered library can produce composition, marker evidence and descriptive program scores. The standard install includes AUCell. For communication, install the signaling extra with python -m pip install -e '.[signaling]' from this checkout. Broad regulation also uses the isolated hdWGCNA and pySCENIC environments and species-matched reference files. Advanced study setup.
For a smaller CPU run, choose explore for QC, clustering and UMAP or abundance for a replicate-aware comparison. Open my_study/run/reports/report.html for the figures and tables. First-study walkthrough.
For a worked comparison, the cells and nuclei example uses matched libraries from four donors and tests cell-type abundance by donor.
![]() |
![]() |
![]() |
| Clusters and their relationships | Differential expression, donor-aware | Differential abundance by cell type |
![]() |
![]() |
![]() |
| Cell-cell communication | Regulatory networks | Expression along a trajectory |
These are drawn by CellQuorum's own figure code from small synthetic data, and each one is pinned by a pixel-exact test, so a code change cannot quietly alter how a figure looks. See the full gallery.
- Check what ran. The report shows which stages completed, warnings from individual methods, and reasons for skipped or failed stages. A stage can complete with a warning about a method or figure it could not produce.
- Donors are the replicates. Differential expression is pseudobulked per donor and abundance tests can be paired by donor, so a few thousand cells from one person do not masquerade as a few thousand independent observations.
- Reproducible and auditable. Every run writes its resolved configuration, plan, backend versions and an index of registered outputs. A resumed run refuses a checkpoint that was made with different settings.
- Installation checks. CI installs the built package into a clean environment and exercises its CLI. The real-data benchmark describes the biological comparisons tested and their limitations.
The core workflow carries one AnnData object through QC, preprocessing, integration, and clustering. Downstream stages reuse its counts, labels, and study design for differential expression, abundance, regulatory networks, communication, and trajectory analysis. Some methods require an R installation, an isolated environment, or reference data. See the backend guide for their requirements.
- Config over code. Every method is chosen in YAML (
integration.method: harmony | scvi), so a dataset is expressed as configuration, not a script. - GPU by default, when available. Normalization, PCA, neighbors, and Leiden route onto
rapids-singlecell/cupywhen a capable CUDA device is present, and fall back to scanpy (CPU) otherwise — with identical output keys either way. - Python + R, transparently. R/Bioconductor methods (edgeR, Milo, propeller, NicheNet, DIALOGUE, SoupX, hdWGCNA) and isolated heavyweight backends (pySCENIC, scCODA, CellOracle) are dispatched behind one interface.
- Fail-loud data contracts. Structural, layer-provenance, and statistical checks reject, for example, raw counts mislabeled as log-normalized.
- Reproducible & auditable. A standardized run directory plus machine-readable provenance (resolved config, plan, backend status, environment/version stamp, artifact manifest).
- Two front doors. A
cellquorum/cqCLI and arun_pipelinePython API.
A run validates the config, plans the enabled stages against detected backends,
bootstraps the run directory, then executes each enabled stage in canonical order,
propagating the updated AnnData downstream:
The diagram is generated from the stage registry (docs/assets/gen_pipeline_diagram.py),
so it always matches the engine: a shared preprocessing → integration → annotation
backbone produces one annotated object, then the four downstream analysis families
fan out in parallel and converge on the run directory. Each stage row names its
config-selectable methods. A stage also skips cleanly when disabled in the config or when its required
inputs or backends are unavailable.
CellQuorum supports Python 3.12 and 3.13, both tested in CI (3.14 is allowed by the package metadata but not yet tested). From a clone:
git clone https://github.com/SecondBook5/cellquorum.git
cd cellquorum
# create an environment (conda/mamba/micromamba recommended)
mamba create -n cellquorum-dev python=3.12 -y
mamba activate cellquorum-dev
# install the package (with dev extras) in editable mode
python -m pip install -e ".[dev]"
# optional: install pre-commit hooks
pre-commit installOptional extras: .[r] (rpy2 bridge for R/Bioconductor methods), .[gpu]
(scvi-tools), .[workflow] (Snakemake, for the multi-hypothesis matrix workflow). The heavyweight and GPU stacks (RAPIDS, pySCENIC, scCODA, CellOracle,
hdWGCNA) live in dedicated environments — see Backends & environments
and envs/README.md. A layered Docker image bakes all of them;
see docs/guide/docker.md.
The first-study walkthrough explains how to choose a study
design and read the report. To preview your saved configuration before a run, use
cellquorum plan -c my_study/config.yaml. You can edit the YAML to add analyses.
# 1. inspect the execution plan for a config (which stages run, on which backends)
cellquorum plan --config configs/config.yaml
# 2. run the full pipeline into a run directory
cellquorum run --config configs/config.yaml --output-dir runs/example_run
# initialize the run directory + provenance without executing stages
cellquorum run --config configs/config.yaml --output-dir runs/example_run --bootstrap-only
# machine-readable output for either command
cellquorum plan --config configs/config.yaml --jsoncellquorum run validates the config, writes provenance, then executes every
enabled stage, threading the AnnData object from stage to stage and recording a
structured success / skip / failure for each.
A dataset and its analysis are described entirely in YAML and validated by a strict
Pydantic schema (extra = "forbid" — unknown keys are rejected). The top-level
sections:
| Section | Configures |
|---|---|
project |
name, organism, species_id |
paths |
data_root, run_root, scratch_root, manifest, output_dir |
input |
input .h5ad path, counts layer, optional subset |
run |
profile, run_id, random_seed, resume, overwrite, logging |
compute |
backend (auto/cpu/gpu), prefer_gpu, fallback_to_cpu, n_jobs |
r |
R/Rscript backend preference, rscript_path, timeout |
report |
HTML / Markdown / PDF report toggles |
stages |
per-stage enable flags |
| per-stage blocks | one block per stage (e.g. integration:, enrichment:) selecting the method and its parameters |
markers, cohort, design, contrasts |
marker panels, obs-key schema, experimental design, named case/control contrasts |
Every section has defaults, so a minimal config is small. A method is chosen per
stage with a method: (or methods:) key inside that stage's block:
project:
name: cellquorum_project
organism: human
species_id: 9606
compute:
backend: auto # prefer GPU when a capable device is present, else CPU
prefer_gpu: true
fallback_to_cpu: true
stages:
qc_evidence: true
preprocessing: true
dimensionality: true
integration: true
clustering: true
annotation: true
differential_expression: true
differential_abundance: true
enrichment: true
integration:
method: harmony # or: scvi
differential_abundance:
method: milo # or: sccoda | propeller | proportion_ttestSee docs/guide/configuration.md for the full section-by-section
reference, configs/generic_pbmc_example.yaml for a runnable single-object example
(point input.h5ad at your data), and configs/config.yaml for the reference
configuration.
Each stage validates its input contract, dispatches to the config-selected method, and records the outcome. Stages marked isolated env run as subprocesses in a dedicated environment; R methods run over the Rscript/rpy2 bridge. Method names link to their projects; licenses and citations are in Third-party software.
| Stage | Methods | Backend |
|---|---|---|
ambient_correction |
SoupX | R |
qc |
MAD/fixed thresholds; Scrublet + scDblFinder doublet consensus; SampleQC joint profiles; Tirosh cell-cycle | Python (+R) |
preprocessing |
PFlog1pPF / log1p-CP10k normalization (layer-tagged) | Python · GPU |
feature_selection |
HVG; Seurat v3; Pearson-residuals/deviance | Python |
dimensionality |
PCA + scree + n_pcs: auto (knee, logged) |
Python · GPU |
integration |
Harmony (CPU); scVI (GPU-gated) | Python · GPU |
clustering |
Leiden + neighbors on the integrated embedding | Python · GPU |
subclustering |
recursive Leiden with sc-SHC significance; CHOIR | Python |
annotation (+ consensus, diagnostics, adjudication) |
marker-vote; CellTypist; passthrough; scDiagnostics | Python (+R) |
reference_mapping |
scArches (atlas-agnostic) | Python |
integration_benchmark |
scIB-style metrics | Python |
population_identity |
population-level identity scoring | Python |
state_scoring |
curated cell-state program scoring — scanpy score_genes + decoupler AUCell (stress/HSP, hypoxia, IFN, senescence, fibrosis) |
Python |
discovery |
de-novo program discovery — consensus NMF (replicate fits + KMeans consensus spectra, cNMF-style) | Python |
composition |
descriptive cell-type composition — overall, per-condition, per-sample proportions + stacked-bar figures | Python |
embeddings |
UMAP + PHATE + PAGA (incl. PAGA-on-UMAP); feature/score overlays with opt-in MAGIC | Python |
differential_expression (+ de_viz) |
donor-aware pseudobulk (edgeR); volcano | R |
differential_state |
donor-paired testing of per-cell state/program scores between conditions | Python |
differential_abundance |
paired arcsin-sqrt t-test; Milo; scCODA; propeller | Python + R + isolated env |
enrichment (+ enrichment_viz) |
GSEA; ORA; GSVA; decoupler activity (CollecTRI/PROGENy) | Python |
coexpression |
hdWGCNA co-expression modules | isolated R env |
grn |
pySCENIC regulons (GRNBoost2 → cisTarget → AUCell) | isolated env |
perturbation |
CellOracle in-silico TF knockouts (ranked target table) | isolated env |
cell_cell_communication |
LIANA consensus; Tensor-cell2cell; NicheNet + MultiNicheNet | Python + R |
multicellular_programs |
DIALOGUE cross-cell-type coordinated programs | R |
ccc_network |
topology (roles + PageRank); Ollivier-Ricci curvature | Python |
ccc_viz |
dotplot / chord / Sankey / curvature-network / summary | Python |
trajectory (+ trajectory_viz) |
scVelo RNA velocity per lineage (velocyto loom ingestion); CellRank fate probabilities; Palantir and diffusion pseudotime; CytoTRACE 2 | Python |
CellQuorum uses layered environments to isolate incompatible dependency stacks. Backend availability is detected at runtime and reported in provenance; an absent backend causes the affected method to skip with a reason, never to crash the run.
Primary environments: cellquorum-core (main runtime + CLI), cellquorum-dev
(adds test/lint/build tooling), cellquorum-gpu (CUDA PyTorch + scvi-tools + RAPIDS),
cellquorum-r (Seurat, zellkonverter, rpy2, anndata2ri).
Isolated backend environments (invoked as subprocesses; names are hardcoded in the dispatch code, so create them exactly as named):
| Environment | Powers | Why isolated |
|---|---|---|
pyscenic_env |
grn (pySCENIC) |
pins numpy/pandas/setuptools incompatible with the core stack |
hdwgcna_env |
coexpression (hdWGCNA) |
R + Seurat/WGCNA stack |
sccoda_env |
differential_abundance (scCODA) |
older scipy/tensorflow pins |
celloracle_env |
perturbation (CellOracle) |
dependency isolation for reproducibility |
scclr |
preprocessing/dimensionality (sparse PFlog1pPF + PCA) |
pins anndata<0.11, Python ≤ 3.13 |
R/Bioconductor methods (edgeR, Milo, propeller, NicheNet, MultiNicheNet, DIALOGUE,
scDiagnostics) run over the Rscript/rpy2 bridge through one shared abstraction
(cellquorum.methods.r_method.RAnalysisMethod). See
docs/guide/backends.md and envs/README.md for
setup and make lock for pinned environments.
Normalization (PFlog1pPF via cupy), PCA, and neighbors + Leiden (via
rapids-singlecell) run on the GPU when rapids-singlecell + cupy are installed
and a CUDA device is present, and fall back to scanpy on CPU otherwise. A shared
compute router gates on real capability — not merely a visible device — and
routed methods produce identical output keys on both paths, so results and data
contracts are path-independent. Set compute.backend: cpu to force CPU.
Before analysis begins, CellQuorum writes machine-readable provenance into
<run>/provenance/:
| File | Purpose |
|---|---|
resolved_config.json |
the validated, fully-resolved runtime configuration |
pipeline_plan.json / stage_plan.csv |
selected stages, ordering and reasons for omissions |
backend_status.json / .csv |
detected backend availability |
planner_warnings.json |
planner warnings |
run_metadata.json |
run identity, paths, profile, seed, environment/version stamp |
artifact_manifest.csv |
index of every generated artifact |
A run directory has a fixed layout:
runs/example_run/
├── figures/ ├── objects/ ├── reports/ ├── results/
├── logs/ ├── provenance/ └── scratch/
Opt-in resume (run.resume: true) skips a side-effect-only stage whose prior
completion fingerprint matches the current input and whose artifacts still exist.
from cellquorum import run_pipeline
result = run_pipeline(
config="configs/config.yaml", # path, dict, or CellQuorumConfig
output_dir="runs/example_run",
)
print(result.context.paths.root) # the run directory
print(result.plan.enabled_stage_names()) # stages the plan enabled
# execution_result is populated when execute=True (the default)
if result.execution_result is not None:
print(result.execution_result.succeeded_stage_names())
print(result.execution_result.skipped_stage_names())run_pipeline accepts a YAML config path, a validated CellQuorumConfig, or a
plain dict, and returns a PipelineRunResult with config, plan, context,
artifacts, and (for executed runs) execution_result.
The cellquorum (alias cq) command exposes:
| Command | Purpose | Key options |
|---|---|---|
cellquorum --version |
print the installed version | |
cellquorum plan |
build and display the execution plan | --config/-c, --json |
cellquorum run |
execute a pipeline run | --config/-c, --output-dir/-o, --bootstrap-only, --json, --quiet/-q |
A separate gen-configs run command expands a hypothesis manifest into
per-(hypothesis, cell_type) configs plus an accounting file:
gen-configs run --manifest hypotheses.yaml --template template.yaml --out-dir generated/flowchart LR
CFG([YAML config]) --> VAL[Pydantic validation]
VAL --> REG[Backend registry]
REG --> PLAN[Execution planner]
PLAN --> BOOT[Run bootstrapper]
BOOT --> EXEC[Registry-driven executor]
EXEC -->|per stage| DC{Input contract valid?}
DC -->|yes| M[Dispatch config-selected method]
DC -->|no| SK[Skip with recorded reason]
M --> THREAD[Thread updated AnnData onward]
REG -.-> PROV[(Provenance)]
PLAN -.-> PROV
EXEC -.-> PROV
The source is organized into top-level packages under src/cellquorum/, with
the pipeline-step packages grouped together under stages/ — one canonical import
path per public thing, no compatibility shims (frozen by
tests/repo/test_old_paths_removed.py). The engine machinery is core (context, planner,
executor, contracts, provenance), config, methods (dispatch + shared
abstractions), and backends; stages holds the 14 step packages
(ambient_correction, qc, preprocessing, clustering, integration,
annotation, comparative, composition, metacells, state_scoring, discovery,
gene_regulation, cell_cell_communication, trajectory); and io, stats, datasets,
visualization, api (the public Python API surface), cli, and utils round out the
public surfaces. The stages/comparative package groups the four "compare groups" analyses
as submodules — differential_expression, differential_abundance, enrichment,
and multicellular_programs. See docs/reference/architecture.md.
Begin with these files (paths below are under src/cellquorum/):
-
Entry point: The CLI lives in
cli/app.py— look for themain()function that launches thecellquorum/cqcommand. The programmatic API lives inapi/pipeline.py—run_pipelineis the main Python entry point. -
Run flow: Read
docs/reference/how-it-works.mdfor a complete file-by-file walkthrough of one run: config validation → backend detection → planning → data loading → stage execution → provenance. It names every file on the control-flow spine. -
Configuration: The
config/models.pyfile defines the entire configuration schema as Pydantic models —CellQuorumConfigis the top-level contract. Seedocs/guide/configuration.mdfor a section-by-section reference. -
Architecture:
docs/reference/architecture.mdexplains the design principles, the execution model, and how the pieces fit together.
One-line run trace: cli/app.py:main → validate config (config/models.py) → build plan (core/planner.py reads core/stage_catalog.py) → load data (io/manifest.py) → execute stages (core/executor.py dispatches to stages/<stage>/stage.py, writes via core/stage_artifact_writer.py) → final context with annotated adata.
pytest # full test suite
pre-commit run --all-files # ruff + hooks
cellquorum plan --config configs/config.yaml # CLI smoke checkGPU-marked tests skip on CPU-only machines and run in cellquorum-gpu; R-marked
tests require an Rscript + Bioconductor backend. Markers: gpu, slow, r,
integration.
docs/index.md— documentation homedocs/guide/tutorial.md— hands-on first run, from a raw.h5adto resultsdocs/reference/architecture.md— engine design and execution modeldocs/guide/configuration.md— configuration referencedocs/guide/backends.md— backends and environmentsdocs/reference/third-party-software.md— the software CellQuorum calls: licenses and citationsdocs/guide/docker.md·docs/guide/snakemake.md— containerized and orchestrated runsdocs/ROADMAP.md— roadmap and backlogCHANGELOG.md
If you use CellQuorum in your research, please cite it (GitHub's "Cite this repository" button reads
CITATION.cff; a BibTeX entry is in
Third-party software) and the methods that
produced your results. Every method in the stage table above links to its project, and
Third-party software lists a citation for each.
CellQuorum is released under the BSD 3-Clause License; see LICENSE. The software it
calls keeps its own licenses, several of them GPL, and one required dependency (leidenalg) is
GPL-3.0-or-later. Third-party software lists each
license and explains what that means if you redistribute a container or use CellQuorum
commercially.





