Skip to content

Bulk MultiQC reporting as a pipeline-installed Python module - #45

Merged
jscgh merged 36 commits into
devfrom
multiqc_bulk_report
Oct 1, 2026
Merged

jscgh merged 36 commits into
devfrom
multiqc_bulk_report

Conversation

@keiran-rowell-unsw

Copy link
Copy Markdown
Member

Implements 'bulk' MultiQC reporting as per proteinfold/issues/439.

Recommendation from Phil Ewels when evaluating contribute into MultiQC modules was instead using setup.py in this specific pipeline whereas MultiQC modules typically work with only one CLI tool (MultiQC/MultiQC#3462)

@github-actions

This comment was marked as off-topic.

@github-actions

github-actions Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit a4a242c

+| ✅ 281 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗  15 tests had warnings |!
Details

❗ Test warnings:

  • files_exist - File not found: conf/igenomes.config
  • files_exist - File not found: conf/igenomes_ignored.config
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • local_component_structure - prepare_boltz_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_alphafold2_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_esmfold_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_colabfold_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - post_processing.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - aria2_uncompress.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_alphafold3_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure

❔ Tests ignored:

✅ Tests passed:

Run details

  • nf-core/tools version 4.0.3
  • Run at 2026-10-01 01:54:55

@keiran-rowell-unsw keiran-rowell-unsw added the enhancement New feature or request label Sep 9, 2026
@keiran-rowell-unsw
keiran-rowell-unsw changed the base branch from master to dev September 9, 2026 06:27
Comment thread main.nf Outdated
Comment thread multiqc_proteinfold/multiqc_proteinfold_config.yml Outdated
Comment thread multiqc_proteinfold/proteinfold.py Outdated
keiran-rowell-unsw and others added 7 commits September 23, 2026 10:28
…line

Implement 'bulk' MultiQC reporting as a pipeline-installed Python module.

- Add/setup module packaging for multiqc_proteinfold - won't be upstreamed into MultiQC as per Phil's comments PR#3462
- Install module from pipeline runtime with Docker linting behavior (escaping PWD)
- Handle pip target/PYTHONPATH and improve install robustness
- Ensure MultiQC YAML packag usable
- Apply related lint/format cleanup (e.g. escape PWD)

rank typo

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

use logger not print

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

remove unused multiqc modules

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

typos

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

makesure user can specify sample merge rules

make nextflow lint happy by using shell substitution not VARS

docker container doesn't like shell substitution without escape chars
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

remove SBF logo and custom links
@jscgh
jscgh force-pushed the multiqc_bulk_report branch from a816184 to 2a402a2 Compare September 23, 2026 00:33
@github-actions

This comment was marked as outdated.

@jscgh
jscgh force-pushed the multiqc_bulk_report branch from 2a402a2 to 5cb1311 Compare September 23, 2026 02:08
@github-actions

This comment was marked as off-topic.

Also add setuptools/wheel to the module env (required by --no-build-isolation,
conda-forge python >=3.12 no longer bundles setuptools)
@github-actions

This comment was marked as off-topic.

@jscgh

jscgh commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Nice simplification moving bulk reporting to a pipeline-local MultiQC plugin.

I've updated the multiqc module to 1.35 (via nf-tools) and added pip/pandas/setuptools/wheel to the module env and rebuilt the containers so the plugin install works (verified end-to-end), and tidied the tests.

@jscgh jscgh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting blank sections of reports running the current branch. I've tried to trace the sources and commented on them.

Comment thread assets/report_template.html
Comment thread multiqc_proteinfold/proteinfold.py Outdated
Comment thread workflows/boltz.nf Outdated
@keiran-rowell-unsw

Copy link
Copy Markdown
Member Author

I hadn't noticed the agent swapped multiqc:1.34 for a more involved multiqc_pandas_pip_setuptools_wheel.

It makes it more explicit, but happy to revert to simplify the install process.

Sort ranked models, make the embedded config JSON injection-safe, de-duplicate model images and fix rendering for the ipTM/ipSAE/chainwise tables.
@github-actions

This comment was marked as off-topic.

@github-actions

This comment was marked as off-topic.

@jscgh jscgh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I can tell this was a huge body of work. 🚀
I was able to rebuild the plugin into a small module with mqc custom-content files, so we can use the native nf-core/multiqc version. This should help with keeping it up to date and changing things locally in the future.
I also added some tests for reports to make it easier to track drift.

@jscgh
jscgh merged commit 0bef7bc into dev Oct 1, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants