Record native Bioconductor null-calibration comparators

.github/workflows/ci.yml

@@ -293,6 +293,12 @@ jobs:
       - name: Check DADA2 CSV assembly
         run: Rscript --vanilla test/unit/test_dada2_tables.R
 
+      - name: Check null-calibration CSV handling
+        run: Rscript --vanilla test/unit/test_null_calibration_csv.R
+
+      - name: Check null-calibration failure annotations
+        run: python3 -m unittest discover -s scripts/tests -p 'test_null_calibration_failures.py' -v
+
       # Saved on failure too, since the packages built before a failure are the ones
       # the next attempt should reuse. Root writes the cache, so it is made readable first.
       - name: Make the renv cache readable

.github/workflows/null-calibration-comparators.yml

@@ -233,6 +233,19 @@ jobs:
           bash scripts/ci/run-with-log.sh "$RUNNER_TEMP/summarise.log" -- \
             Rscript --no-init-file bench/null_calibration/summarise.R
 
+      # A test that produced no usable p-value leaves its reason only in the
+      # per-cell CSV, which is an artifact a reader may not be able to open.
+      # The distinct reasons are therefore published as annotations too.
+      - name: Report any test that could not run
+        if: always()
+        run: |
+          set -euo pipefail
+          shopt -s nullglob
+          for csv in bench/null_calibration/results/raw/pvalues_*.csv; do
+            cell="$(basename "$csv" .csv)"
+            python3 bench/null_calibration/report_failures.py "$csv" "$cell"
+          done
+
       - name: Publish the summaries as annotations
         if: always()
         run: |

bench/null_calibration/REPORT.md

@@ -1,15 +1,18 @@
 # Null-calibration bench report (issue #193)
 
-This report records the one run that has actually been executed, superseding the
-reduced 2026-10-22 run in this file's history. Anything not listed here has not
-been run. The committed summary CSVs in `results/`
-(`calibration_summary.csv`, `power_summary.csv`) are the source of truth for
-every number below; the per-cell run records are `results/provenance_summary.csv`
-(generated from the per-cell provenance files, not hand-written). The raw
-p-value files under `results/raw/` are gitignored and reproducible from the
-recorded seeds and arguments.
-
-## Run identity
+This report combines the original 2026-10-23 WebR simulation study and a
+supplemental native-R/Bioconductor comparator run completed on 2026-10-24.
+The reduced 2026-10-22 run remains superseded. The committed
+`calibration_summary.csv`, `power_summary.csv`, and `provenance_summary.csv`
+cover the original WebR cells. The native comparator
+run is preserved separately in
+`results/native_comparator_summary_run_40091522843.csv` and
+`results/native_comparator_provenance_run_40091522843.csv`, captured from the
+successful check-run annotations for [Actions run 40091522843](https://github.com/hyperpolymath/MetaManifold-WebUI/actions/runs/40091522843).
+Raw p-value files remain outside Git; the run's `null-calibration-raw` artifact
+is retained through 2026-11-23.
+
+## Original WebR baseline run identity
 
 - **Date:** 2026-10-23 (UTC, per-cell provenance records).
 - **Executor:** WebR 0.6.0 (R 4.6.0, wasm32) through `webr_run.mjs`, because the
@@ -21,12 +24,12 @@ recorded seeds and arguments.
 - **Inputs pinned by digest:** `bench/null_calibration/calibration.R`
   md5 `2717e709ff718e57e602eeaff2410e71`; extracted `_FIT_R` block md5
   `cc9efdfdd6e4efe2f9a393ba4d178ce0`.
-- **Not available in this sandbox:** `glmGamPoi` and `edgeR` (compiled
-  Bioconductor packages). `wald_trend`, `glmgampoi_ql` and `edger_ql` were
-  therefore not run; their code paths in `calibration.R` are unexecuted here.
-  §"What was not run" gives the commands for a native R/Bioconductor run.
-- **Tests run in every cell:** `wald_ml` (the production `_FIT_R` block,
-  `da_use_gp = FALSE`), `lrt` and `ql_f` (both from one full `glm.nb` fit, plus
+- **Not available in the WebR baseline:** `glmGamPoi` and `edgeR` (compiled
+  Bioconductor packages). The baseline did not run `wald_trend`,
+  `glmgampoi_ql` or `edger_ql`; the supplemental native-R run below measures
+  them for original/TSS/null cells at n=4 and n=12.
+- **Tests run in every baseline cell:** `wald_ml` (the production `_FIT_R`
+  block, `da_use_gp = FALSE`), `lrt` and `ql_f` (both from one full `glm.nb` fit, plus
   one intercept-only fit shared by `lrt`).
 - **Seeds:** `seed0 = 20261020`; per-sim seed =
   `(seed0 + cell_key × 1000003 + sim) mod 2³¹−1`, where `cell_key` is a hash of
@@ -167,40 +170,38 @@ Readings:
   |t| > 2.45. The calibration at that size is bought with power, and no measured
   option is a free lunch at 3–4 samples per group.
 
-## What was not run, and why
+## Native R/Bioconductor comparator run (issue #261)
+
+The comparator lane completed successfully in [Actions run 40091522843](https://github.com/hyperpolymath/MetaManifold-WebUI/actions/runs/40091522843), triggered by tag `null-calibration-run4-2026-10-24` at measured commit `bc435feb4614c29f4339fe578caba0c938b6e4c9`. It was configured for 1,000 null simulations × 40 taxa at original shape/TSS offset for n=4 and n=12 per group, using the same per-cell seed scheme as the WebR study. Its summary and provenance are preserved in `results/native_comparator_summary_run_40091522843.csv` and `results/native_comparator_provenance_run_40091522843.csv`.
+
+The run used the final two post-PR fixes from that tag lineage: the loud failure-reason reporting commit `17ce140` and `test_de` `sort_by` fix `bc435fe`. They were cherry-picked onto this work branch as `2599317` and `8784a11`. The measured file digests are recorded from the run itself: `calibration.R` md5 `12a90679f74d2e1a9ab28cec643b5e76`; extracted production `_FIT_R` md5 `cc9efdfdd6e4efe2f9a393ba4d178ce0`.
+
+After measurement, this PR refactored package-result normalization and cell-warning logic in response to Sonar findings, and hardened CSV serialization plus resume validation (schema and cell-key checks) so package error notes remain valid data. The comparator workflow now reads failure rows with a CSV-aware parser. These post-run changes do not alter estimator calls, simulation seeds, or formulas. At this PR revision the working branch's `calibration.R` md5 is `3ed22ba8b04f695d531a45925abbc13f`, not the measured run's `12a90679f74d2e1a9ab28cec643b5e76`; the results below remain tied to the measured tag/code digest above.
 
-`glmGamPoi` and `edgeR` are compiled Bioconductor packages and are not
-installable in this sandbox (no native R, and the WebAssembly package repository
-does not carry them). Their bench code paths are therefore unexecuted here:
+| n/group | test | package | summary rows (`n_attempted`) | p-values (failed fits) | α=0.05 (Wilson 95% CI) | verdict |
+|---:|---|---|---:|---:|---|---|
+| 4 | `glmgampoi_ql` | glmGamPoi 1.22.0 | 39,993 | 39,904 (89) | **0.0525** (0.0503–0.0547) | anti-conservative, narrowly |
+| 4 | `edger_ql` | edgeR 4.8.2 | 39,993 | 39,904 (89) | **0.0687** (0.0663–0.0713) | anti-conservative |
+| 4 | `wald_trend` | glmGamPoi 1.22.0 | 39,993 | 39,904 (89) | **0.0838** (0.0811–0.0865) | anti-conservative |
+| 12 | `glmgampoi_ql` | glmGamPoi 1.22.0 | 40,000 | 40,000 (0) | 0.0499 (0.0478–0.0520) | not flagged |
+| 12 | `edger_ql` | edgeR 4.8.2 | 40,000 | 40,000 (0) | **0.0712** (0.0687–0.0737) | anti-conservative |
+| 12 | `wald_trend` | glmGamPoi 1.22.0 | 40,000 | 40,000 (0) | **0.0764** (0.0739–0.0791) | anti-conservative |
 
-```sh
-Rscript -e 'renv::restore()'   # native R with the lockfile
-Rscript bench/null_calibration/calibration.R \
-  --shapes=original --offsets=tss --n-per=4,12 --sims=1000 --taxa=40 \
-  --tests=glmgampoi_ql,edger_ql,wald_trend
-Rscript bench/null_calibration/summarise.R
-```
+The `n_attempted` summary field is `nrow(d)`: it counts rows present in the per-test output, not the 40,000 planned taxon-simulation positions. The `taxa=40` setting is the raw design size, not an observed post-filter testing denominator. At n=4, 39,904 p-values plus 89 failed fits account for 39,993 rows; seven of the 40,000 positions in the configured 1,000 × 40 design are absent from the summary, and their disposition is unresolved. Rejection rates use the 39,904 reported p-values; do not assign the seven absent positions to either successes or failures. At n=12, all 40,000 rows are reported as p-values with no failed fits.
 
-They skip *loudly* when absent (a stdout line plus a provenance record), so the
-committed results never silently imply they ran. Issue #193's own measurement
-remains the reference for glmGamPoi's QL F-test (0.046 and 0.048 at 4 and 12
-per group); this bench adds a dispersion-scaled QL F test that can be run
-without Bioconductor. Both land within about a point of nominal, against 0.14
-and 0.075 for the production Wald test at the same sizes.
+The verdict rule remains the one declared above: a Wilson 95% lower bound above 0.05 is anti-conservative; an interval containing 0.05 is “not flagged”, not proof of exact calibration. The glmGamPoi QL result is just above nominal at n=4 (its lower bound is 0.0503) and not flagged at n=12. edgeR QL is anti-conservative at both sizes. `wald_trend` is the glmGamPoi dispersion-trend Wald comparator, **not** glmGamPoi's QL test.
+
+Provenance: native R 4.5.0 (2025-04-11), x86_64 Linux, MASS 7.3.65, glmGamPoi 1.22.0, edgeR 4.8.2; `seed0=20261020`; `calibration.R` md5 `12a90679f74d2e1a9ab28cec643b5e76`; extracted production `_FIT_R` md5 `cc9efdfdd6e4efe2f9a393ba4d178ce0`. The run tested all six methods in its two cells; the full 12-row summary is retained in the run-specific CSV. The raw p-value artifact remains on Actions with 30-day retention, through 2026-11-23.
+
+## What remains unmeasured
+
+The successful native run fills issue #261's requested glmGamPoi QL and edgeR QL null-calibration cells (and also records `wald_trend`) for n=4 and n=12 only. Those package comparators have not been measured here at other group sizes, shapes, offsets, or under the power/alternative design. The original 16-cell WebR study still uses 400 simulations for the shape/offset/power cells; that limitation is explicit above. The native null-only result does not establish power, FDR under alternatives, broad robustness, or a production-default recommendation.
 
 ## Checklist mapping (#193)
 
-Done: the 1,000-sim direct replication at six group sizes with Wilson intervals
-at α = 0.01/0.05/0.10 and KS tests; constant, sparse and unequal-library shapes;
-RLE as well as TSS offsets; the LRT and an additional quasi-likelihood F test
-paired on identical datasets; power and BH-FDR at \|log2FC\| = 1; the bench lives
-in `bench/null_calibration/` and never runs in CI.
-
-Not done here: glmGamPoi QL F, edgeR QL F and the production `glmGamPoi`
-dispersion-trend Wald option, all of which need native R + Bioconductor
-(commands above); the full 1,000-sim count on the shape/offset/power cells
-(400 each — sufficient to resolve a 4-point difference at α = 0.05, and stated
-above rather than implied).
+Done: the 1,000-sim direct replication at six group sizes with Wilson intervals at α = 0.01/0.05/0.10 and KS tests; constant, sparse and unequal-library shapes; RLE as well as TSS offsets; the LRT and tagwise quasi-likelihood F test paired on identical datasets; power and BH-FDR at |log2FC| = 1; and the native-R glmGamPoi QL, edgeR QL, and `wald_trend` null cells at n=4 and n=12. The bench remains opt-in and never runs in ordinary CI.
+
+Not done here: power/alternative calibration or other shapes/offsets for the native Bioconductor comparators; the full 1,000-simulation count on the original shape/offset/power cells (400 each — sufficient to resolve a 4-point difference at α = 0.05, as stated rather than implied); and any independent scientific review or upstream default decision.
 
 ## Consequence for the production caveat
 

bench/null_calibration/UPSTREAM_DRAFT.md

@@ -1,9 +1,10 @@
 # Draft upstream report: null calibration of the `glm.nb` Wald test
 
-*Status: draft prepared downstream for review before anything is filed
-upstream. Every number is from `REPORT.md` in this directory, which defines
-exactly what ran, where, and what did not run. This replaces the earlier draft
-based on the reduced 2026-10-22 run.*
+*Status: draft prepared downstream for maintainer review; it has not been filed
+upstream. It includes the completed native comparator run #40091522843.
+`REPORT.md` and the run-specific summary/provenance CSVs define exactly what ran,
+where, and what remains unmeasured. This replaces the earlier draft based on
+the reduced 2026-10-22 run and does not recommend an automatic default change.*
 
 ## Summary
 
@@ -28,6 +29,21 @@ Two cheaper alternatives, measured on identical datasets:
   0.055 at 4, 0.058–0.060 through 8, 0.056 at 20. It remains ~1 point above
   nominal, not at nominal.
 
+## Native Bioconductor comparators
+
+The successful [native-R Actions run 40091522843](https://github.com/hyperpolymath/MetaManifold-WebUI/actions/runs/40091522843) used the same seeded null designs at original shape/TSS, n=4 and n=12, configured for 1,000 simulations × 40 taxa per cell (R 4.5.0, glmGamPoi 1.22.0, edgeR 4.8.2):
+
+| n/group | comparator | summary rows (`n_attempted`) | p-values (failed fits) | α=0.05 (Wilson 95% CI) | verdict |
+|---:|---|---:|---:|---|---|
+| 4 | glmGamPoi QL F (`glmgampoi_ql`) | 39,993 | 39,904 (89) | **0.0525** (0.0503–0.0547) | narrowly anti-conservative |
+| 4 | edgeR QL F (`edger_ql`) | 39,993 | 39,904 (89) | **0.0687** (0.0663–0.0713) | anti-conservative |
+| 4 | glmGamPoi trend Wald (`wald_trend`) | 39,993 | 39,904 (89) | **0.0838** (0.0811–0.0865) | anti-conservative |
+| 12 | glmGamPoi QL F (`glmgampoi_ql`) | 40,000 | 40,000 (0) | 0.0499 (0.0478–0.0520) | not flagged |
+| 12 | edgeR QL F (`edger_ql`) | 40,000 | 40,000 (0) | **0.0712** (0.0687–0.0737) | anti-conservative |
+| 12 | glmGamPoi trend Wald (`wald_trend`) | 40,000 | 40,000 (0) | **0.0764** (0.0739–0.0791) | anti-conservative |
+
+“Not flagged” means the Wilson interval includes 0.05; it is not proof of exact calibration. The summary's `n_attempted` field is computed as `nrow(d)`, the rows present in each per-test output; the `taxa=40` setting is the raw design size, not an observed post-filter testing denominator. At n=4, the summary reports 39,904 p-values and 89 failed fits (39,993 output rows total); seven positions from the configured 40,000 taxon-simulation design are absent from the summary, and their disposition is unresolved. Rejection rates use the 39,904 reported p-values; do not assign the seven absent positions to successes or failures or treat the 40,000 design count as an observed testing denominator. At n=12, the summary reports 40,000 p-values and no failed fits. The native comparator lane tested nulls only: it did not measure power, BH-FDR under alternatives, other sample sizes, or other shapes/offsets. The complete 12-row output and per-cell provenance are retained in `results/native_comparator_summary_run_40091522843.csv` and `results/native_comparator_provenance_run_40091522843.csv`.
+
 The inflation is not specific to one data shape: constant dispersion, sparse
 zero-heavy tables, unequal library sizes and RLE offsets all reproduce it
 (4/group Wald 0.112–0.142; 12/group 0.070–0.081). RLE is not a remedy; at
@@ -47,9 +63,11 @@ Under a power design (25% of taxa at |log2FC| = 1, 400 sims), BH at 0.05:
 | 12 | QL F (`ql_f`) | 0.433 | 0.063 |
 
 The default's apparent power advantage at small n is an artefact of an
-uncontrolled FDR. The QL F test is the only measured option whose FDR is near
-nominal, but at 4/group it is nearly powerless after BH because its F reference
-has only six residual degrees of freedom (`F(1,6)` ≈ 5.99, |t| > 2.45).
+uncontrolled FDR. Among the three methods included in this power design, the
+tagwise QL F test is closest to nominal FDR, but at 4/group it is nearly
+powerless after BH because its F reference has only six residual degrees of
+freedom (`F(1,6)` ≈ 5.99, |t| > 2.45). The native Bioconductor comparator run
+was null-only and did not measure their power or FDR under alternatives.
 
 ## Options for the upstream maintainer's consideration
 
@@ -61,17 +79,22 @@ has only six residual degrees of freedom (`F(1,6)` ≈ 5.99, |t| > 2.45).
    but still anti-conservative everywhere, slower, and it fails more often at
    tiny n. It is a mitigation, not a calibration fix; the numbers above quantify
    the trade-off.
-3. **Quasi-likelihood F test below a group-size threshold.** Closest to nominal
-   at every size and the only option with near-nominal BH FDR, but it changes
-   critical values substantially at 3–4 samples per group and costs almost all
-   BH power there. glmGamPoi's own QL F-test (the calibrated path in the
-   downstream issue) is the version worth considering rather than a tagwise
-   reimplementation, and it was not measured in this sandbox (see `REPORT.md`
-   §"What was not run" for the native-R command).
-
-The downstream recommendation is **1 now, 2 and 3 as an upstream decision**.
-A default change would need the glmGamPoi QL F row measured on native R first;
-this bench provides the code path and the protocol for it.
+3. **Consider glmGamPoi's QL F test below a group-size threshold.** The native
+   null run places it closest to nominal among the two Bioconductor QL tests:
+   0.0525 at 4/group (Wilson 95% CI 0.0503–0.0547, narrowly anti-conservative)
+   and 0.0499 at 12/group (0.0478–0.0520, not flagged). edgeR QL is higher at
+   0.0687 and 0.0712. The separate `wald_trend` row is a glmGamPoi
+   dispersion-trend Wald test, not its QL test; the production fixed-θ Wald
+   path must not be described as glmGamPoi QL. These comparator cells are
+   null-only: they do not measure power/BH behaviour, other sample sizes, or
+   other shapes/offsets. The tagwise `ql_f` power results therefore cannot be
+   transferred to glmGamPoi QL. The measured row makes option 3 discussable,
+   not an automatic default recommendation.
+
+The downstream recommendation remains **1 now, 2 and 3 as an upstream
+maintainer decision**. The native run fills the requested QL calibration cells,
+but it does not establish a default threshold or quantify the power trade-off
+for glmGamPoi QL. No production default change is proposed by this draft.
 
 ## Reproducing
 
@@ -83,6 +106,10 @@ Rscript bench/null_calibration/calibration.R --shapes=original --offsets=tss \
 Rscript bench/null_calibration/summarise.R
 ```
 
-The committed `results/` CSVs carry every rate and interval quoted above;
-`results/provenance_summary.csv` carries the executor, the package versions and
-the input digests for each cell.
+The committed `results/` CSVs carry the baseline rates and intervals. The
+native comparator output and provenance are recorded separately in
+`results/native_comparator_summary_run_40091522843.csv` and
+`results/native_comparator_provenance_run_40091522843.csv`; the source is [Actions
+run 40091522843](https://github.com/hyperpolymath/MetaManifold-WebUI/actions/runs/40091522843),
+tag `null-calibration-run4-2026-10-24`. The raw Actions artifact is retained
+through 2026-11-23.