Added support for Cleanifier tool: cleanifier/download, cleanifier/index and cleanifier/filter by LeonHornich · Pull Request #12360 · nf-core/modules

LeonHornich · 2026-07-14T22:55:18Z

With this PR I want to add support for the Cleanifier tool for species contamination removal. There are 3 added modules: download, index and filter.

Unfortunately I am still experiencing some issues with nf-test snapshot stability. If anyone has an idea on how to fix them i'd very much appreciate that. Of course other feedback is also highly appreciated.

PR checklist

Closes #XXX

famosab · 2026-07-15T11:12:53Z

I would suggest to split this up into one PR by subtool / command :) regarding the md5sum instability I will add a comment

famosab

I added a few hints :)

famosab · 2026-07-15T11:13:42Z

We want the ext.args to be present in the main.nf.test file. That makes everything more readable in one go. See the module specifications for more info.

famosab · 2026-07-15T11:14:16Z

You can run

nextflow lint -format -sort-declarations -spaces 4 -harshil-alignment

on this file to clean this up nicely.

famosab · 2026-07-15T11:14:46Z

Please remove this file :)

famosab · 2026-07-15T11:14:54Z

@@ -0,0 +1,5 @@
+process {
+    withName: CLEANIFIER_INDEX {
+        ext.args = "-k 31"


We want the ext.args to be present in the main.nf.test file. That makes everything more readable in one go. See the module specifications for more info.

famosab · 2026-07-15T11:19:38Z

                """
            }
        }

        then {
            assertAll(
                { assert process.success },
-                { assert snapshot(sanitizeOutput(process.out)).match() }
+                { assert snapshot(process.out).match() }


I would just first assert the txt content of the filter output and then we can probably see why the md5sum is instable

Suggested change

{ assert snapshot(process.out).match() }

{ assert snapshot(

file(process.out.filter[0][1]).name,

process.out.info,

process.out.findAll { key, val -> key.startsWith("versions") },

).match() }

Edit: I updated to just use the name

I played around with the index tests. From my understanding the core issue is that cleanifier/index generates a index file using a probabilistic cuckoo filter. These are not deterministic and therefore differ between snapshots.

Since this doesn't seem avoidable i am now only assessing the files existence (via file path) in the most recent commit.

While this seems to improve my snapshots locally, I now get an error while linting:

nf-core modules lint cleanifier (nextflow_env) ,--./,-. ___ __ __ __ ___ /,-._.--~\ |\ | |__ __ / ` / \ |__) |__ } { | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' nf-core/tools version 4.0.2 - https://nf-co.re INFO Linting modules repo: '.' INFO Linting module: 'cleanifier' ╭─ [✗] 1 Module Test Failed ───────────────────────────────────────────────────╮ │ ╷ ╷ │ │ Module name │ File path │ Test message │ │╶────────────────────┼───────────────────────────┼───────────────────────────╴│ │ cleanifier/index │ modules/nf-core/cleanifi… │ test_snap_versions: │ │ │ │ versions not found in │ │ │ │ snapshot file │ │ ╵ ╵ │ ╰──────────────────────────────────────────────────────────────────────────────╯ ╭───────────────────────╮ │ LINT RESULTS SUMMARY │ ├───────────────────────┤ │ [✔] 166 Tests Passed │ │ [!] 0 Test Warnings │ │ [✗] 1 Test Failed │ ╰───────────────────────╯

How would you usually handle this according to nf-core guidelines? Thank you :)

You need to add this line:

process.out.findAll { key, val -> key.startsWith("versions") },

because otherwise the versions are not asserted

famosab · 2026-07-16T09:16:09Z

@@ -35,7 +35,7 @@ nextflow_process {
                { assert snapshot(
                    path(process.out.index[0][1]).getFileName().toString(),
                    path(process.out.info[0][1]).getFileName().toString(),


Was the info output also instable? Otherwise we couls just keep it

Yes, the .info file also seems unstable. My assumption is that is stores data on the probabilistic cuckoo filter. Since these are unstable to begin with, this would make the .info files unstable as well :)

famosab · 2026-07-16T09:17:04Z

Is there now way to test the real download? Maybe for a small subsection?

The download command essentially allows us to download pre-computed index for human data. This is based on the t2t human reference genome. While there are 2 index files you can download, they are 7 and 14GB large. There is no option to download only parts of that. Since this sounded like too large of a testing case i only implemented strip cases for the cleanifier/download module.

famosab · 2026-07-16T09:18:02Z

+    setup {
+        run("CLEANIFIER_INDEX") {
+            script "../../index/main.nf"
+            config "./nextflow.config"


Suggested change

config "./nextflow.config"

duplicated to above

famosab · 2026-07-16T09:20:12Z

+        echo "" | gzip > ${prefix}.clean_keep.fastq.gz
+        ${keep_host ? "echo \"\" | gzip > ${prefix}.clean_filter.fastq.gz" : ""}
+        """
+    }
+    else {
+        """
+        echo "" | gzip > ${prefix}.clean_1_keep.fastq.gz
+        echo "" | gzip > ${prefix}.clean_2_keep.fastq.gz
+        ${keep_host ? "echo \"\" | gzip > ${prefix}.clean_1_filter.fastq.gz" : ""}
+        ${keep_host ? "echo \"\" | gzip > ${prefix}.clean_2_filter.fastq.gz" : ""}
+        """


We usually do not need the "" in the echo statement :)

Suggested change

echo "" | gzip > ${prefix}.clean_keep.fastq.gz

${keep_host ? "echo \"\" | gzip > ${prefix}.clean_filter.fastq.gz" : ""}

"""

}

else {

"""

echo "" | gzip > ${prefix}.clean_1_keep.fastq.gz

echo "" | gzip > ${prefix}.clean_2_keep.fastq.gz

${keep_host ? "echo \"\" | gzip > ${prefix}.clean_1_filter.fastq.gz" : ""}

${keep_host ? "echo \"\" | gzip > ${prefix}.clean_2_filter.fastq.gz" : ""}

"""

echo | gzip > ${prefix}.clean_keep.fastq.gz

${keep_host ? "echo | gzip > ${prefix}.clean_filter.fastq.gz" : ""}

"""

}

else {

"""

echo | gzip > ${prefix}.clean_1_keep.fastq.gz

echo | gzip > ${prefix}.clean_2_keep.fastq.gz

${keep_host ? "echo | gzip > ${prefix}.clean_1_filter.fastq.gz" : ""}

${keep_host ? "echo | gzip > ${prefix}.clean_2_filter.fastq.gz" : ""}

"""

If I do that the linter is complaining, I will be defaulting back to using the "".

nf-core modules lint cleanifier (nextflow_env) ,--./,-. ___ __ __ __ ___ /,-._.--~\ |\ | |__ __ / ` / \ |__) |__ } { | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' nf-core/tools version 4.0.2 - https://nf-co.re INFO Linting modules repo: '.' INFO Linting module: 'cleanifier' ╭─ [✗] 1 Module Test Failed ───────────────────────────────────────────────────╮ │ ╷ ╷ │ │ Module name │ File path │ Test message │ │╶────────────────────┼───────────────────────────┼───────────────────────────╴│ │ cleanifier/filter │ modules/nf-core/cleanifi… │ test_stub_gzip_syntax: │ │ │ │ Stub gzip files must use │ │ │ │ 'echo "" | gzip >' syntax. │ │ │ │ Invalid patterns found: │ │ │ │ echo | gzip > │ │ │ │ ${prefix}.clean_keep.fastq │ │ │ │ .gz │ │ ╵ ╵ │ ╰──────────────────────────────────────────────────────────────────────────────╯ ╭───────────────────────╮ │ LINT RESULTS SUMMARY │ ├───────────────────────┤ │ [✔] 166 Tests Passed │ │ [!] 0 Test Warnings │ │ [✗] 1 Test Failed │ ╰───────────────────────╯

added 3 new modules supporting Cleanifier tool

d5fbec1

LeonHornich added Ready for Review new module Adding a new module and removed Ready for Review labels Jul 15, 2026

famosab reviewed Jul 15, 2026

View reviewed changes

LeonHornich added 2 commits July 16, 2026 09:34

applied suggestions and tried fixing snapshot instability

87fefb2

corrected cleanifier/index nf-test versions testing

cbfcef4

famosab reviewed Jul 16, 2026

View reviewed changes

LeonHornich added 2 commits July 16, 2026 18:35

some more refinements

46473a3

reverted echo changes in cleanifier filter stub test

5c86c1b

Uh oh!

Conversation

LeonHornich commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

famosab commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

famosab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

famosab Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

famosab Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

famosab Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LeonHornich commented Jul 14, 2026 •

edited

Loading

famosab commented Jul 15, 2026 •

edited

Loading

famosab Jul 15, 2026 •

edited

Loading

famosab Jul 16, 2026 •

edited

Loading

famosab Jul 16, 2026 •

edited

Loading