Sg/backend improvements

.github/workflows/docs.yml

@@ -12,7 +12,7 @@ jobs:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
         with:
-          python-version: "3.10.20"
+          python-version: "3.12"
       - name: Install scalr requirements
         run: |
           pip install -r requirements.txt

.github/workflows/publish-to-pypi.yml

@@ -15,7 +15,7 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v5
       with:
-        python-version: "3.10.20"
+        python-version: "3.12"
     - name: Install pypa/build
       run: >-
         python3 -m

.github/workflows/run_isort.yml

@@ -10,10 +10,10 @@ jobs:
 
     steps:
     - uses: actions/checkout@v2
-    - name: Set up Python 3.10.20
+    - name: Set up Python 3.12
       uses: actions/setup-python@v2
       with:
-        python-version: "3.10.20"
+        python-version: "3.12"
     - name: Install isort
       run: pip install isort
     - name: Run isort

.github/workflows/run_pytest.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.10.20"]
+        python-version: ["3.11", "3.12", "3.13", "3.14"]
 
     steps:
       - uses: actions/checkout@v3
@@ -22,6 +22,6 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+          python -m pip install -e ".[dev]"
       - name: Running test cases
         run: pytest -v -s

.gitignore

@@ -0,0 +1,24 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Local run artifacts (quickstart/README commands write here by default)
+nohup.out
+example_data/
+models/
+
+# IDE / OS
+.vscode/
+.idea/
+.DS_Store
+
+# Docs build output
+_build/

.venv/pyvenv.cfg

@@ -0,0 +1,5 @@
+home = /bin
+include-system-site-packages = false
+version = 3.12.3
+executable = /usr/bin/python3.12
+command = /bin/python3 -m venv /home/saurabh/infocusp_git/scaLR/.venv

CHANGELOG.md

@@ -0,0 +1,46 @@
+# Changelog
+
+All notable changes to scaLR are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+
+## [Unreleased]
+
+## [1.2.0] - 2026-10-13
+
+### Added
+- Hugging Face Hub integration: `scalr.models.push`/`pull` and `scalr models push`/`pull` to publish and download model artifacts (`scalr/models.py`).
+- Dataset drift detection: `scalr.detect_drift` / `scalr drift` to check whether a query dataset's distribution has shifted from a model's training data (`scalr/drift.py`).
+- Explainability API: `scalr.explain` / `scalr explain` for gene-attribution explanations of model predictions (`scalr/explain.py`).
+- Streaming/chunked prediction (`streaming=True`, `chunk_size=...`) in `scalr.annotate`/`AnnotationModel.predict` for bounded-memory inference on large datasets.
+- Support for custom/user-defined loss functions via the `name`/`params` builder pattern (`scalr/nn/loss/_loss.py`).
+- `scalr` CLI: `train`, `annotate`, `explain`, `evaluate`, `drift`, and `models {list,info,push,pull}` subcommands, plus a `[dev]` extra and Development setup docs in the README.
+- `.gitignore` for local run artifacts (`.venv/`, `__pycache__/`, `example_data/`, `models/`, build output).
+
+### Changed
+- Feature scoring in `FeatureExtractionPipeline` parallelized across feature subsets.
+- `DataIngestionPipeline`: `train_val_test` config checks consolidated into a single `_validate_train_val_test_config` method, run up front instead of being interleaved with split generation.
+
+### Fixed
+- Gene alignment when a dataset has duplicate gene names (`scalr/genes.py`, `scalr/feature/feature_subsetting.py`).
+- Multi-worker config propagation during chunked preprocessing.
+- `anndata` bumped past a broken 0.11.x chunked-read regression; fixed a `pydeseq2` `np.matrix` incompatibility in pseudobulk DGE (`scalr/analysis/dge_pseudobulk.py`).
+- Disabled `AnnCollection` dtype harmonization, which was corrupting chunked `.h5ad` reads.
+
+### Removed
+- Dropped Python 3.10 support (`requires-python`, CI matrix, README). The `anndata>=0.12` floor needed for the chunked-read fix above requires Python >=3.11, so 3.10 installs could no longer resolve a compatible `anndata` version.
+
+### Performance
+- Lazy-import scanpy/SHAP-backed submodules (`scalr/analysis`, `scalr/feature/scoring`) to reduce base import time.
+
+### Documentation
+- `README.md` restructured into a concise entry point, with detailed content split out into linked guides: [docs/guide_simple_api.md](docs/guide_simple_api.md), [docs/guide_cli.md](docs/guide_cli.md), [docs/guide_pipeline.md](docs/guide_pipeline.md), and [docs/tutorials.md](docs/tutorials.md).
+- Added [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, tests, and formatting instructions.
+- Added Sphinx autodoc coverage for the entire simple-API layer (`docs/scalr.simple_api.rst`: `api`, `artifact`, `benchmark`, `calibration`, `cli`, `doublet`, `drift`, `explain`, `feature_stability`, `genes`, `hierarchy`, `leakage`, `metrics`, `model_card`, `models`, `refinement`, `result`, `validation`), which previously had no published documentation at all.
+- Fixed broken/orphaned Sphinx toctree entries (`docs/modules.rst`, `docs/scalr.rst`) and resolved a duplicate-object-description warning on `scalr.result.PredictionResult`.
+- Added the previously-missing SHAP analysis tutorial to the tutorials list ([docs/tutorials.md](docs/tutorials.md)).
+- Fixed stale `anndata==0.10.9` pins in `dge.ipynb`, `gene_recall_curve.ipynb`, `normalization.ipynb`, and `batch_correction.ipynb` (incompatible with the `anndata>=0.12` floor above), and a stale feature-branch image URL in `scalr_pipeline.ipynb`/`scalr_pipeline_local_run.ipynb`.
+
+## [1.1.0]
+
+Baseline prior to this changelog's introduction. See `git log` for history before this point.

CLAUDE.md

@@ -0,0 +1,90 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this is
+
+scaLR (Single-cell analysis using Low Resource) is a Python library/pipeline for scRNA-seq cell-type classification, feature selection, and biomarker analysis, designed to work on large datasets with limited compute. It has two layers:
+
+1. **Simple library API** (`scalr.train`, `scalr.annotate`, `scalr.load_model`, `scalr.validate`, `scalr.align_genes`) — an inference-first API added on top of the engine for quick, config-free use. See `scalr/api.py`, `scalr/artifact.py`, `scalr/result.py`.
+2. **Configuration-driven pipeline** (`pipeline.py` + a YAML config) — the original, full-featured research pipeline: data ingestion/splitting/preprocessing → feature extraction → model training → evaluation & downstream analysis. This remains the primary way to run large, customized experiments.
+
+Both layers share the same underlying engine in `scalr/` (models, dataloaders, trainers, losses, splitters, preprocessors, feature scorers/selectors, analysis modules).
+
+## Commands
+
+Install (editable, for development):
+```bash
+pip install -r requirements.txt
+pip install -e .
+```
+CPU is the default `torch` install; for CUDA, install a CUDA-specific `torch` build per the [PyTorch instructions](https://pytorch.org/get-started/locally/) before or after. `requirements.txt` uses bounded version ranges so pip can select wheels for the active Python version while avoiding major-version compatibility breaks.
+
+`pip install -e .` also registers the `scalr` console script (`scalr validate|train|annotate|evaluate|models ...`, see `scalr/cli.py`), a thin wrapper around the simple API below.
+
+Run the full config-driven pipeline:
+```bash
+python pipeline.py --config /path/to/config.yaml -l -m
+```
+(`-l`/`--log` enables experiment logging, `-m`/`--memoryprofiler` enables memory profiling.)
+
+Tests (pytest, no separate test directory — test files live next to the code as `test_*.py`):
+```bash
+pytest                          # run everything
+pytest -v -s                    # verbose, as CI does
+pytest scalr/test_api.py        # a single test file
+pytest scalr/test_api.py::test_train_returns_annotation_model_with_metrics   # a single test
+```
+
+Formatting/import order (both enforced in CI, config in `.style.yapf` / `.isort.cfg`, both based on the `google` style, 80-col):
+```bash
+yapf -ir --style .style.yapf .
+isort . --settings-path .isort.cfg
+```
+`pre-commit` is configured (`.pre-commit-config.yaml`) to run both automatically.
+
+Docs (Sphinx, published to GitHub Pages on push to `main`):
+```bash
+sphinx-build docs _build
+```
+
+## Architecture
+
+### Config-driven pipeline (`scalr/*_pipeline.py`, driven by `pipeline.py`)
+
+`pipeline.py` reads one YAML config and runs whichever top-level sections are present (sections can be omitted/commented out to skip stages):
+
+1. **`data`** → `DataIngestionPipeline` (`data_ingestion_pipeline.py`): splits data into train/val/test (via `scalr/data/split/`), applies preprocessing (`scalr/data/preprocess/`), generates label mappings.
+2. **`feature_selection`** → `FeatureExtractionPipeline` (`feature_extraction_pipeline.py`): trains on feature subsets batch-wise, scores features (`scalr/feature/scoring/`: linear-weight or SHAP-based), and selects top-k features (`scalr/feature/selector/`).
+3. **`final_model`**/training → `ModelTrainingPipeline` (`model_training_pipeline.py`): builds model/optimizer/loss/callbacks and trains via a `Trainer` (`scalr/nn/trainer/`).
+4. **`analysis`** → `EvalAndAnalysisPipeline` (`eval_and_analysis_pipeline.py`): evaluation metrics/classification report, gene/biomarker analysis, and pluggable downstream analyses (`scalr/analysis/`: DGE via pseudobulk or linear mixed models, ROC-AUC, gene recall curves, heatmaps).
+
+Each stage returns an updated config dict so the full resolved config (including defaults) gets written back out — this is how `config/README.md` params map to code.
+
+### The `name`/`params` builder pattern
+
+Almost every pluggable component (models, dataloaders, loss functions, splitters, preprocessors, feature scorers/selectors, callbacks, analysers) is instantiated from a config dict of the form `{name: <ClassName>, params: {...}}` via `scalr.utils.build_object` (`scalr/utils/misc_utils.py`), which looks up `<ClassName>` in the given module, merges `params` over the class's `get_default_params()`, and constructs it. Each such subpackage exposes a `build_*` convenience function next to its base class (e.g. `build_model` in `scalr/nn/model/_model.py`, `build_preprocessor` in `scalr/data/preprocess/_preprocess.py`, `build_splitter`, `build_analyser`, `build_dataloader`, `build_loss_fn`). When adding a new implementation of any of these, follow the existing sibling classes' pattern: subclass the module's base class, implement its abstract methods, and implement `get_default_params()`.
+
+### Data handling
+
+- Input is `.h5ad` (AnnData) or a directory of chunked `.h5ad` files, read via `scalr.utils.read_data`/`write_data` (`scalr/utils/file_utils.py`), which returns an `AnnCollection` for chunked directories to avoid loading everything into memory.
+- `adata.X` is expected to already be normalized (`log1p`, range 0–10) for the pipeline convention; `adata.obs` holds cell metadata including the classification target column; `adata.var` holds gene names.
+- Chunkwise writing/transformation (`write_chunkwise_data`) is the mechanism used throughout to keep peak memory bounded regardless of dataset size — this is central to the "low resource" design goal, so avoid introducing full-dataset `.toarray()`/dense materialization in new code.
+- `scalr.data.split` splitters (`StratifiedSplitter`, `GroupSplitter`, `StratifiedGroupSplitter`) matter for scientific validity: `GroupSplitter`/`StratifiedGroupSplitter` keep a `stratify` column (e.g. `donor_id`) confined to a single split to prevent train/test leakage — prefer these over plain stratified splitting whenever a donor/patient/sample identifier is available.
+
+### Simple API layer (`scalr/api.py`, `scalr/artifact.py`, `scalr/result.py`, plus one module per capability)
+
+This is a newer, self-contained layer built on top of the same engine (`scalr.nn.model`, etc.) but bypassing the config/pipeline machinery for simple in-memory workflows. Each capability lives in its own top-level module and is wired into `AnnotationModel`/`api.py` as an optional parameter, not a separate code path:
+
+- `scalr.train(adata, labels_key, group_key=...)` does leakage-safe splitting (`scalr/leakage.py`), class-weighted training, temperature-scaling calibration (`scalr/calibration.py`), and macro/weighted-F1 + balanced-accuracy evaluation (`scalr/metrics.py`), returning an `AnnotationModel`. Pass `taxonomy={fine_label: broad_label, ...}` to enable hierarchical annotation.
+- `AnnotationModel.save()`/`scalr.load_model()` (`scalr/artifact.py`) persist/restore a **self-contained model artifact directory** (`manifest.json`, `model.pt`, `model_config.json`, `label_mapping.json`, `features.json`, `preprocessing.json`, `calibration.json`, `metrics.json`, `taxonomy.json` if set, and an auto-generated `README.md` model card via `scalr/model_card.py`) — no external config needed to reload and run inference. `scalr.load_model` also resolves a bare name via the local model registry (`scalr/models.py`, `scalr.models.list/info/register`; `download()` intentionally raises — no hosted hub yet).
+- `scalr.annotate(adata, model=...)` / `AnnotationModel.predict()` validate the input (`scalr/validation.py`), align its genes to the model's expected feature list (`scalr/genes.py`, zero-filling missing genes), run inference, and return a typed `PredictionResult` (`scalr/result.py`) with calibrated `confidence`, `entropy`, `margin`, and an `is_unknown` open-set abstention flag — always aligned to `adata.obs_names` (`PredictionResult.write_to_adata` refuses to write if names don't match, to prevent silently misaligned predictions). Optional `predict()`/`annotate()` params layer on: `level="broad"` (aggregate via taxonomy, `scalr/hierarchy.py`), `cluster_refinement="auto"|<obs_key>` (confidence-weighted per-cluster majority vote, `scalr/refinement.py` — always preserves the pre-refinement labels in `result.metadata["raw_labels"]`), and `flag_doublets=True` (sets `result.is_possible_doublet`, a heuristic screening signal from `scalr/doublet.py`, not a validated doublet call).
+- `scalr.compute_feature_stability(adata, labels_key, ...)` (`scalr/feature_stability.py`) repeats feature selection over resampled subsets (via `sklearn.LogisticRegression`, independent of the DNN training loop) and reports per-gene selection frequency and Jaccard stability — for judging whether a selected biomarker set is a stable signal or a split-specific artifact.
+- `scalr.run_benchmark(datasets, labels_key, ...)` (`scalr/benchmark.py`) runs train+predict per dataset and reports both quality (macro-F1, balanced accuracy) and cost (wall time, cells/sec, peak RSS via `resource.getrusage`) together, plus environment metadata — never scale one dataset's numbers without the other.
+- `scalr/cli.py` exposes `validate`/`train`/`annotate`/`evaluate`/`models list`/`models info` as the `scalr` console script (registered in `pyproject.toml` under `[project.scripts]`); it's a thin argparse wrapper around this same API, not a separate implementation.
+
+When extending this layer, keep it usable without any YAML config — pass all such new functionality through function parameters. This layer works with in-memory `AnnData` directly rather than the chunked/config-driven flow used by the pipeline, so it's not currently intended for atlas-scale/out-of-core datasets.
+
+### Logging
+
+Two logger classes in `scalr/utils/logger.py`: `FlowLogger` prints high-level stage progress to stdout (use this for anything user-facing/visible by default). `EventLogger` writes detailed logs to a file and is **silent by default** (`NullHandler`) unless a `filepath` or `stdout=True` is given — don't use it where output should be visible without extra configuration.

CONTRIBUTING.md

@@ -0,0 +1,38 @@
+# Contributing to scaLR
+
+## Development setup
+
+```
+python -m pip install -r requirements.txt
+python -m pip install -e ".[dev]"
+pre-commit install
+```
+
+## Tests
+
+pytest, test files live next to the code as `test_*.py`, no separate test directory:
+
+```
+pytest                    # run everything
+pytest -v -s              # verbose, as CI does
+pytest scalr/test_api.py  # a single test file
+```
+
+## Formatting / import order
+
+Both enforced in CI (config in `.style.yapf` / `.isort.cfg`), and run automatically by `pre-commit`:
+
+```
+yapf -ir --style .style.yapf .
+isort . --settings-path .isort.cfg
+```
+
+## Docs
+
+Sphinx, published to GitHub Pages on push to `main`:
+
+```
+sphinx-build docs _build
+```
+
+See [CHANGELOG.md](CHANGELOG.md) for a history of notable changes.