Problem: segmentation fault

[feldroop]

Hello, I am currently working on a thorough benchmark for Rust FM-Indices. I am including the library AWRY, which uses sufr to construct the suffix array. I got a segmentation fault when running the benchmark and traced it down to sufr functions. I believe the unsafety happens in the slice_to_u8_vec function. I am running the benchmark on a machine that supports avx 512 with rustflags = ["-C", "target-cpu=native"]. The whole error message was:

Exception has occurred: Signal
signal SIGSEGV: address not mapped to object (fault address: 0x7fffc3495000)

The benchmark is still fluid work inprogess, but you can find it here, if you want to attempt reproducing the issue.

[kyclark]

Felix, can you provide a minimal failing condition for the error? E.g., a test input and the exact command you're running?

Also, I'm having problems compiling your benchmark using Rust 1.90.0:

error[E0433]: failed to resolve: could not find `FmIndexFlat64` in `genedex`
   --> src/main.rs:248:44
    |
248 |         Library::GenedexFlat64 => genedex::FmIndexFlat64::<G>::run_benchmark(config),
    |                                            ^^^^^^^^^^^^^ could not find `FmIndexFlat64` in `genedex`

error[E0433]: failed to resolve: could not find `FmIndexCondensed64` in `genedex`
   --> src/main.rs:249:44
    |
249 |         Library::GenedexCond64 => genedex::FmIndexCondensed64::<G>::run_benchmark(config),
    |                                            ^^^^^^^^^^^^^^^^^^ could not find `FmIndexCondensed64` in `genedex`

error[E0433]: failed to resolve: could not find `FmIndexCondensed512` in `genedex`
   --> src/main.rs:250:45
    |
250 |         Library::GenedexCond512 => genedex::FmIndexCondensed512::<G>::run_benchmark(config),
    |                                             ^^^^^^^^^^^^^^^^^^^ could not find `FmIndexCondensed512` in `genedex`

error[E0603]: module `text_with_rank_support` is private
  --> src/genedex_bench.rs:5:14
   |
 5 | use genedex::text_with_rank_support::TextWithRankSupport;
   |              ^^^^^^^^^^^^^^^^^^^^^^ private module
   |
note: the module `text_with_rank_support` is defined here
  --> /home/kyclark/.cargo/git/checkouts/genedex-c0c37eef1e10b90d/877f62c/src/lib.rs:44:1
   |
44 | mod text_with_rank_support;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^

Some errors have detailed explanations: E0433, E0603.
For more information about an error, try `rustc --explain E0433`.

[feldroop]

Hi Ken,

Sorry about that, I was just changing up some things and working with a local version of the genedex library. The latest commit of the benchmark repo should compile now.

Sadly, the only input I have is not really minimal, because I only tested the whole benchmark on the large input files. You can run the benchmark by first downloading the input data (linked in the README) and then creating the following file/folder structure inside the benchmark repo:

data/
    hg38.fna <- renamed downloaded reference genome
    reads.fastq <- renamed downloaded reads from SRA

Just as a heads up, the script will create modified versions of the reference genome as input, with total disk space of over 10 GB.

You can then run: cargo run --release -- awry -i hg38 -s 4 -d 10 -l 50 to hopefully reproduce the issues.

[kyclark]

Does it matter if I use the GCA_000001405.27_GRCh38.p12_genomic.fna or GCF_000001405.38_GRCh38.p12_genomic.fna file for the hg38.fna? I'm able to run it now, but it's taking a very long time due to the highly repetitive nature of HG38. How long does it take for the error to manifest for you?

[feldroop]

I'm using the GCF... file, I'm not sure if the other one triggers the same behavior. Yes, it's taking a while (at least 3-4 hours, without debugger). The issue seems to happen in one of the later steps of the algorithm. Here is the output:

------------------------------ starting benchmark for Awry ------------------------------
Current memory usage after reading texts: 3382.0 MB
Scanned for runs of Ns in 2.437263296s
Selected 10239 pivots in 11.058799ms
Wrote 3298431341 unsorted suffixes to partitions in 4773.176660431s
Sorted 3298431341 suffixes in 1024 partitions (avg 3221124) in 8287.653403959s
Segmentation fault

I believe you are also using rayon? The -t parameter of the benchmark tool sets the number of threads for rayon, with 1 being the default. You could try and see if the error also happens when using more threads.