Corrections to the CHANGELOG and report no longer reading the trace

CHANGELOG.md

@@ -25,16 +25,18 @@ Initial release of nf-core/deepmutscan, created with the [nf-core](https://nf-co
 - DMS library QC per library: count and count-per-coverage heatmaps, sorted count distributions with LogDiff (variants without counts drawn at a floor derived from the data and counted separately; LogDiff not reported when more than 10% of variants have no counts), sliding-window coverage and count profiles (`--sliding_window_size`, `--aimed_cov`), and sequencing-depth rarefaction on a geometric depth grid (`--run_seqdepth`)
 - Optional fitness estimation (`--fitness`): merged count tables, experimental design, synonymous wildtype proxy selection, default log-ratio fitness with replicate rescaling and summary statistics, replicate correlation plots and fitness heatmap
 - Optional fitness estimation with DiMSum (`--dimsum`) and mutscan edgeR / limma (`--mutscan`)
-- Minimum observation count for a variant to be retained (`--min_counts`, default 1), applied during variant counting; the sequencing-error correction runs on the pre-threshold counts
+- Minimum number of reads a variant needs in every `input` replicate to receive a fitness estimate (`--min_counts`, default 10), applied in the same way by the default estimator, DiMSum and mutscan; variant counting, the sequencing-error correction and the library QC always use all reads
+- Pseudocount for variants that drop out of an `output` replicate (`--output_pseudocount`, default 1; 0 gives dropouts no fitness value)
 - Optional interactive 3D variant effect inspection tool built from a user-supplied wildtype structure (`--pdb`)
-- Self-contained, all-in-one run report (`deepmutscan_report.html`) embedding QC, error-correction, fitness, MultiQC and run statistics
+- Self-contained, all-in-one run report (`deepmutscan_report.html`) embedding QC, error-correction, fitness and MultiQC results, with links to Nextflow's execution report and timeline
+- Docker and Singularity images (amd64 and arm64) and conda lock files for all modules, built from each module's `environment.yml`; DiMSum and mutscan are amd64 only for now
 - `test` profile on a 50,000 read-pair subsample of a GID1A nicking-mutagenesis GluePCA experiment, with three nf-test pipeline tests (all options; without fitness, error correction, SeqDepth and structure; stub run), and `test_full` profile on the complete dataset (ENA PRJEB110196); test data and the PDB structure are hosted on [nf-core/test-datasets](https://github.com/nf-core/test-datasets/tree/deepmutscan)
 
 ### `Dependencies`
 
 | Dependency    | Version               |
 | ------------- | --------------------- |
-| `fastqc`      | 0.12.1                |
+| `fastqc`      | 0.13.0                |
 | `fq`          | 0.12.0                |
 | `multiqc`     | 1.35                  |
 | `bwa`         | 0.7.19                |

assets/summary_report/summary_report_template.html

@@ -1319,56 +1319,16 @@
             (s) =>
               `<tr><td>${esc(s.id)}</td><td>${s.ec ? fmt(s.ec.n_variants) : "–"}</td><td>${s.ec ? fmt(s.ec.total_corrected) : "–"}</td></tr>`,
           ).join("")}</tbody></table></div></div>`;
-          // Run statistics from the Nextflow execution trace (Overview), when it was available.
-          const RS = (DATA.run || {}).stats;
-          if (RS && RS.processes && RS.processes.length) {
-            const dur = (s) => {
-              s = Math.round(s);
-              if (s < 60) return s + " s";
-              const m = Math.floor(s / 60),
-                ss = s % 60;
-              if (m < 60) return m + "m" + (ss ? " " + ss + "s" : "");
-              return Math.floor(m / 60) + "h " + (m % 60) + "m";
-            };
-            const sz = (b) => {
-              if (!b) return "–";
-              const u = ["B", "KB", "MB", "GB", "TB"];
-              let i = 0;
-              while (b >= 1000 && i < u.length - 1) ((b /= 1000), i++);
-              return (b < 10 ? b.toFixed(1) : Math.round(b)) + " " + u[i];
-            };
-            const t = RS.total;
-            h += `<div class="panel"><h2>Run statistics ${info(
-              "Per-process compute from Nextflow's execution trace at report time (excludes this report and the trailing version collation). CPU time sums task runtimes; CACHED tasks reused earlier results.",
-            )}</h2>`;
-            h += `<div class="tiles">${[
-              ["Tasks", fmt(t.tasks)],
-              ["Cached", fmt(t.cached)],
-              ["CPU time", dur(t.cpu_time)],
-              ["Peak memory", sz(t.peak_rss)],
-            ]
-              .map(([k, v]) => `<div class="tile"><div class="k">${esc(k)}</div><div class="v">${esc(v)}</div></div>`)
-              .join("")}</div>`;
-            h += `<div class="tblwrap"><table><thead><tr><th>Process</th><th>Tasks</th><th>Cached</th><th>CPU time</th><th>Mean %CPU</th><th>Peak RSS</th></tr></thead><tbody>${RS.processes
-              .map(
-                (p) =>
-                  `<tr><td>${esc(p.process)}</td><td>${fmt(p.tasks)}</td><td>${fmt(p.cached)}</td><td>${dur(
-                    p.cpu_time,
-                  )}</td><td>${p.cpu_pct == null ? "–" : fmt(p.cpu_pct) + "%"}</td><td>${sz(p.peak_rss)}</td></tr>`,
-              )
-              .join("")}</tbody></table></div>`;
-            // Straight into Nextflow's own richer report + timeline, which sit next to this file in
-            // pipeline_info/. Relative links, so they resolve when the report is opened from the results
-            // folder (the location it is published to).
-            const sfx = (DATA.run || {}).trace_report_suffix;
-            if (sfx)
-              h += `<p class="desc" style="margin-top:10px">Nextflow's own outputs, next to this file: <a href="pipeline_info/execution_report_${esc(
-                sfx,
-              )}.html" target="_blank" rel="noopener" style="color:var(--accent);font-weight:600">execution report ↗</a> · <a href="pipeline_info/execution_timeline_${esc(
-                sfx,
-              )}.html" target="_blank" rel="noopener" style="color:var(--accent);font-weight:600">timeline ↗</a></p>`;
-            h += `</div>`;
-          }
+          // Run statistics: Nextflow's own execution report and timeline, which sit next to this file in
+          // pipeline_info/. Linked rather than parsed, because the trace is still being written while this
+          // report is built. Relative links resolve when the report is opened from the results folder.
+          const sfx = (DATA.run || {}).trace_report_suffix;
+          if (sfx)
+            h += `<div class="panel"><h2>Run statistics</h2><p class="desc">Task runtimes, CPU and memory usage of this run are in Nextflow's own reports, next to this file: <a href="pipeline_info/execution_report_${esc(
+              sfx,
+            )}.html" target="_blank" rel="noopener" style="color:var(--accent);font-weight:600">execution report ↗</a> · <a href="pipeline_info/execution_timeline_${esc(
+              sfx,
+            )}.html" target="_blank" rel="noopener" style="color:var(--accent);font-weight:600">timeline ↗</a></p></div>`;
           // README: what this file is and how it maps onto the results folder. The report is meant as
           // the way in - a reader should be able to orient here and then find the underlying files.
           const ec = r.error_correction && r.error_correction !== "none";

docs/output.md

@@ -198,7 +198,7 @@ This tool projects the mean per-residue fitness, coverage, counts, counts per co
 
 </details>
 
-The report brings all the important metrics and visual results of a run together in one file; it can be opened in any browser and shared without any accompanying files. Features are a complete `nf-core` run overview with parameters and execution statistics, interactive sequencing and library QC plots per library, the error-correction report, the fitness results and heatmap (with `--fitness`), the DiMSum and mutscan results (if run), a 3D structure view (with `--pdb`), the MultiQC report, and the citations of all the tools and major dependencies that were used in the run.
+The report brings all the important metrics and visual results of a run together in one file; it can be opened in any browser and shared without any accompanying files. Features are a complete `nf-core` run overview with parameters and links to Nextflow's execution report and timeline, interactive sequencing and library QC plots per library, the error-correction report, the fitness results and heatmap (with `--fitness`), the DiMSum and mutscan results (if run), a 3D structure view (with `--pdb`), the MultiQC report, and the citations of all the tools and major dependencies that were used in the run.
 
 ![Run overview](images/run_overview.png)
 

docs/usage.md

@@ -207,7 +207,7 @@ When `--fitness` is set and a wildtype 3D structure is supplied via `--pdb <stru
 
 ### 9. Reporting
 
-`nf-core/deepmutscan` writes a single, self-contained `deepmutscan_report.html` to the top of the results directory. It embeds the library QC plots, the error-correction report, the fitness results (and DiMSum/mutscan output, if run), the [MultiQC](https://multiqc.info) report, run statistics and the citations of all tools that were used, so it can be shared as a single file.
+`nf-core/deepmutscan` writes a single, self-contained `deepmutscan_report.html` to the top of the results directory. It embeds the library QC plots, the error-correction report, the fitness results (and DiMSum/mutscan output, if run), the [MultiQC](https://multiqc.info) report, links to Nextflow's execution report and timeline, and the citations of all tools that were used, so it can be shared as a single file.
 
 ## Updating the pipeline
 

modules/local/visualization/summary_report/templates/build_summary_report.py

@@ -489,78 +489,6 @@ def structure_data(pdb_path, wt_seq, fitness):
             "identity": round(identity, 4), "n_residues": len(residues), "pdb": pdb_text}
 
 
-def _dur_s(s):
-    """Nextflow-formatted duration ('7m 37s', '746ms', '1h 2m') to seconds."""
-    s = str(s).strip()
-    if not s or s == "-":
-        return 0.0
-    tot = 0.0
-    for num, unit in re.findall(r"([\\d.]+)\\s*(ms|s|m|h|d)", s):
-        tot += float(num) * {"ms": 0.001, "s": 1, "m": 60, "h": 3600, "d": 86400}[unit]
-    return tot
-
-
-def _size_b(s):
-    """Nextflow-formatted size ('3 GB', '4.9 MB') to bytes."""
-    m = re.match(r"([\\d.]+)\\s*(B|KB|MB|GB|TB)", str(s).strip(), re.I)
-    if not m:
-        return 0.0
-    return float(m.group(1)) * {"B": 1, "KB": 1e3, "MB": 1e6, "GB": 1e9, "TB": 1e12}[m.group(2).upper()]
-
-
-def run_stats(path):
-    """Per-process compute stats from Nextflow's execution trace.
-
-    The trace is written continuously, so at report time it holds every task finished so far - i.e.
-    everything but the report itself and the version collation that follows it. That is the whole
-    compute-heavy body of the run, which is what the page is about; it is labelled 'as of report
-    generation' rather than pretended to be the final total. CACHED tasks are counted separately
-    because they consumed no compute this run.
-    """
-    df = pd.read_csv(path, sep="\\t", dtype=str, keep_default_na=False)
-    if "name" not in df.columns:
-        return None
-
-    def proc(n):
-        return re.sub(r"\\s*\\(.*\\)\\s*\$", "", n).split(":")[-1]  # drop "(sample)" and the module path
-
-    rows = {}
-    for _, r in df.iterrows():
-        p = proc(r.get("name", ""))
-        d = rows.setdefault(p, {"n": 0, "cached": 0, "cpu_time": 0.0, "rss": 0.0, "cpu": [], "submit": ""})
-        d["n"] += 1
-        if r.get("status", "") == "CACHED":
-            d["cached"] += 1
-        d["cpu_time"] += _dur_s(r.get("realtime", ""))
-        d["rss"] = max(d["rss"], _size_b(r.get("peak_rss", "")))
-        # earliest submission time, so the table can read top-to-bottom in run order. The trace timestamp
-        # is "YYYY-MM-DD HH:MM:SS.mmm", which sorts lexically = chronologically, so no parsing is needed.
-        sub = r.get("submit", "")
-        if sub and (not d["submit"] or sub < d["submit"]):
-            d["submit"] = sub
-        c = r.get("%cpu", "").replace("%", "").strip()
-        try:
-            d["cpu"].append(float(c))
-        except ValueError:
-            pass
-    procs = [{
-        "process": p, "tasks": d["n"], "cached": d["cached"],
-        "cpu_time": round(d["cpu_time"], 1), "peak_rss": round(d["rss"]),
-        "cpu_pct": round(sum(d["cpu"]) / len(d["cpu"])) if d["cpu"] else None,
-        "submit": d["submit"],
-    } for p, d in rows.items()]
-    # chronological: first-submitted process first (falls back to name for any row with no timestamp)
-    procs.sort(key=lambda x: (x["submit"] or "~", x["process"]))
-    status = df["status"] if "status" in df.columns else pd.Series([], dtype=str)
-    total = {
-        "tasks": int(df.shape[0]),
-        "cached": int((status == "CACHED").sum()),
-        "cpu_time": round(sum(p["cpu_time"] for p in procs), 1),
-        "peak_rss": round(max((p["peak_rss"] for p in procs), default=0)),
-    }
-    return {"processes": procs, "total": total}
-
-
 def software_versions(path):
     """Parse pipeline_info's software versions YAML into {process: {package: version}}.
 
@@ -709,11 +637,6 @@ def sample(sid):
         elif kind == "versions":               # pipeline_info software-versions YAML
             versions = software_versions(path)
 
-        elif kind == "trace":                  # Nextflow execution trace -> per-process compute stats
-            try:
-                run["stats"] = run_stats(path)
-            except Exception as exc:  # a malformed/partial trace must never sink the whole report
-                print(f"WARNING: could not parse execution trace: {exc}")
 
     # Coverage needed for every residue outcome to reach --aimed_cov counts, assuming all 21 outcomes
     # are equally represented. Same expression as the reference line in global_position_biases_cov.R;

workflows/deepmutscan.nf

@@ -453,14 +453,6 @@ workflow DEEPMUTSCAN {
     ch_report_assets = ch_report_assets.mix(
         ch_collated_versions.map { f -> [['versions', 'run', 'versions.yml'], f] }
     )
-    // Nextflow's own execution trace drives the Overview run-statistics table. It is still being
-    // appended to as the report is built, so it captures every compute-heavy task (all but the report
-    // task itself) - the page labels it as of report generation. checkIfExists is off because the file
-    // is engine-managed, not a process output.
-    ch_report_assets = ch_report_assets.mix(
-        Channel.fromPath("${outdir}/pipeline_info/execution_trace_${params.trace_report_suffix}.txt", checkIfExists: false)
-            .map { f -> [['trace', 'run', 'execution_trace.txt'], f] }
-    )
 
     // Sorting keeps the manifest and the staged files in one deterministic, matching order.
     def ch_summary_input = ch_report_assets