fibertools pipeline breaking at exclude_from_shuffle

[edna-stewart]

Hello! I am using the fibertools pipeline (using pixi, like pixi run fire --configfile config.yaml ) to ultimately call accessible peaks on m6A-treated DNA. My sample was sequenced using ONT.

I have rerun this code a few times and it always stalled at the same step - 'exclude_from_shuffle' or step 71. Before running this pipeline, I filtered my BAM for highly confident modifications (with an accuracy filter of 0.9) and used fibertools add-nucleosomes.

Using fibertools 0.1.2 with linux-64.

My config file looks like:

> ref: /home/ubuntu/work/mm9/Sequence/WholeGenomeFasta/genome.fa
> ref_name: mm9
> manifest: config.tbl
> ont: True

Any guidance on how to get the pipeline to work would be helpful. Thank you so much!
Below is the relevant piece of the log file indicating the error:

[Wed Dec  3 21:57:07 2025]
Finished job 5.
71 of 202 steps (35%) done
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr1.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr10.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr11.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr12.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr13.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr14.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr15.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr16.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr17.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr18.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr19.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr2.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr3.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr4.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr5.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr6.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr7.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr8.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chr9.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chrM.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chrX.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-chrY.fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/GMP_KO_Hia5_mm9-v0.1.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-fiber-locations.bed.gz.
Removing temporary output temp/GMP_KO_Hia5_mm9/coverage/v0.1-fiber-locations.bed.gz.tbi.
Select jobs to execute...
Execute 1 jobs...

[Wed Dec  3 21:57:08 2025]
localrule exclude_from_shuffle:
    input: temp/GMP_KO_Hia5_mm9/coverage/filtered-for-coverage/v0.1-fiber-locations.bed.gz, /home/ubuntu/work/mm9/Sequence/WholeGenomeFasta/genome.fa.fai
    output: results/GMP_KO_Hia5_mm9/additional-outputs-v0.1/coverage/exclude-from-shuffles.bed.gz
    jobid: 4
    reason: Missing output files: results/GMP_KO_Hia5_mm9/additional-outputs-v0.1/coverage/exclude-from-shuffles.bed.gz; Input files updated by another job: temp/GMP_KO_Hia5_mm9/coverage/filtered-for-coverage/v0.1-fiber-locations.bed.gz
    wildcards: sm=GMP_KO_Hia5_mm9, v=v0.1
    threads: 4
    resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/7c596f604f23a453e334598e67001a40_
It looks as though you have less than 3 columns at line 305 in file stdin.  Are you sure your files are tab-delimited?
[Wed Dec  3 21:58:20 2025]
Error in rule exclude_from_shuffle:
    jobid: 4
    input: temp/GMP_KO_Hia5_mm9/coverage/filtered-for-coverage/v0.1-fiber-locations.bed.gz, /home/ubuntu/work/mm9/Sequence/WholeGenomeFasta/genome.fa.fai
    output: results/GMP_KO_Hia5_mm9/additional-outputs-v0.1/coverage/exclude-from-shuffles.bed.gz
    conda-env: /home/ubuntu/FIRE/.snakemake/conda/7c596f604f23a453e334598e67001a40_
    shell:
        

        (             bedtools genomecov -bga -i temp/GMP_KO_Hia5_mm9/coverage/filtered-for-coverage/v0.1-fiber-locations.bed.gz -g /home/ubuntu/work/mm9/Sequence/WholeGenomeFasta/genome.fa.fai | awk '$4 == 0';             less          )             | cut -f 1-3             | bedtools sort             | bedtools merge             | bgzip -@ 4         > results/GMP_KO_Hia5_mm9/additional-outputs-v0.1/coverage/exclude-from-shuffles.bed.gz
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job exclude_from_shuffle since they might be corrupted:
results/GMP_KO_Hia5_mm9/additional-outputs-v0.1/coverage/exclude-from-shuffles.bed.gz

[mrvollger]

Have you investigated this error message at all?

It looks as though you have less than 3 columns at line 305 in file stdin.  Are you sure your files are tab-delimited?

If there was no luck there, then I think the only way for me to debug is for you to share the data and config files, or to share

temp/GMP_KO_Hia5_mm9/coverage/filtered-for-coverage/v0.1-fiber-locations.bed.gz

[Yanqi-Wang-epigen]

I met with the same problem. Here are my v0.1-fiber-locations.bed.gz

ChatGPT said it's the less in "awk '$4 == 0'; less“ problem. But I don't think so.

v0.1-fiber-locations.bed.gz