Modernizing .NET Bio for .NET 8/10, a few concurrency fixes, and an offer to help maintain

[ataumutozsoy]

Hi @evolvedmicrobe,

I'm a bioinformatics software engineer and I'd like to help bring .NET Bio up to date. I've done a first round of work on a fork and wanted to check with you before opening a PR.

Branch: https://github.com/ataumutozsoy/bio/tree/modernize/net10

What the branch does

• Targets: the libraries target netstandard2.0, net8.0 and net10.0. netstandard2.0 is kept, so .NET Framework users are not affected. The public API and the NuGet package IDs are unchanged.
• Tests: the test projects run on net8.0 and net10.0, since netcoreapp2.0 can no longer run. Across the three test projects all 1,984 tests pass or are skipped on both frameworks, on Windows and Linux.
• CI: Travis is replaced with GitHub Actions, which builds and tests on Linux, Windows and macOS.

Bugs found and fixed along the way

1. Padena gave nondeterministic results. DeBruijnNode keeps all eight extension "invalid" flags in one byte and updates it without a lock. SimplePathContigBuilder.ExcludeAmbiguousExtensions runs in parallel and sets bits on neighbouring nodes too, so concurrent updates to the same node could be lost. As a result contigs were dropped, wrongly merged, or TraceSimplePath threw. On the small-reads test data, about 5% of assemblies gave a different result. After the fix, 2,000 consecutive runs gave identical output.
2. PAMSAM guide trees depended on thread timing. HierarchicalClusteringParallel searched for the closest pair in parallel while writing shared fields, and passed each computed distance through a shared field. A wrong value could end up in the distance matrix.
3. Parsers needed write access to their input. They opened files with FileMode.Open, which requests read/write access, so parsing read-only files failed.
4. The BAM parser and formatter ignored the return value of Stream.Read.
5. 21 GenBank P2 negative tests never ran the parser. The result was never enumerated, the data paths were not resolved, and Assert.Fail was swallowed by a catch-all. 18 of them now pass for real. For the other 3 inputs the parser accepts malformed data; I marked those tests [Ignore] for now.

Each fix is a separate, focused commit.

Found but not changed yet

These change results, so I'd like your opinion first:

• DeBruijnGraph: node.KmerCount <= 255 is always true for a byte, so k-mer counts wrap to 0 at 256 on high-coverage data.
• HierarchicalClusteringParallel.UpdateNearestColumn never updates min, so the nearest distance ends up as float.MaxValue.
• DanglingLinksPurger discards the result of Union(...) (it should be UnionWith).

Questions

1. Would you be willing to review and merge a PR with these changes? I can also split it into smaller PRs if you prefer.
2. Longer term, would you consider adding me as a co-maintainer, including co-ownership of the NetBio.* packages on NuGet? That would let me publish a proper release (3.1 or 4.0). Next I'd like to work on:
   • nullable annotations
   • faster Span<T>-based FASTA/FASTQ/SAM parsers, with benchmarks
   • porting the command line tools to .NET 10

Thanks for creating .NET Bio and keeping it available all these years.

[ataumutozsoy]

A quick update: while benchmarking the parsers I found a correctness bug, and I have performance results.

1. FASTA alphabet detection bug (likely affects real genome files)

Two bugs interact here:

• RnaAlphabet.ValidateSequence and ProteinAlphabet.ValidateSequence loop while i < length instead of i < offset + length.
• FastAParser passes an end index where AutoDetectAlphabet expects a length.

They partly cancel each other out, but for multi-line FASTA records:

• A DNA record with an ambiguity symbol such as N on any line after the first is silently parsed as a protein sequence. Wrapped genome FASTA files with runs of N hit exactly this case.
• Invalid symbols after the first line of RNA and protein records are not detected, even when the alphabet is set explicitly.

The fix, with regression tests, is in ataumutozsoy@61d2428

2. Parser performance

I added a BenchmarkDotNet project and optimized FastAParser and FastQParser without changing the public API. Both columns run on .NET 10; "Before" is the current code.

Benchmark	Before	After	Speedup	Allocated
FASTA, one 10 Mbp sequence	134.5 ms	6.4 ms	20.9×	77.4 → 37.6 MB
FASTA, 20,000 × 1 kbp sequences	266.0 ms	9.4 ms	28.4×	126.4 → 27.3 MB
FASTQ, 100,000 × 150 bp reads	222.8 ms	23.2 ms	9.6×	200.7 → 75.6 MB

Where the time went:

• Most of it was a Dictionary<byte, byte> lookup for every symbol in ValidateSequence. It is now a lookup table, and a vectorized SearchValues scan on .NET 8+.
• Most of the rest was decoding every line into a string and encoding it back to bytes. The parsers now read lines as bytes.

The new byte line reader behaves like StreamReader: the same line breaks, byte order marks, UTF-16/32 input and non-ASCII handling. I checked this two ways:

• fuzz tests that compare it with StreamReader;
• a differential run of 20,000 parse results, including error messages, against the previous implementation. The results were identical.

Branch: https://github.com/ataumutozsoy/bio/tree/perf/parsers. It builds on modernize/net10, so I'd send it as a follow-up PR after the first one.

3. Found, not fixed yet

FastASequencePositionParser.Parse() reads the same stream through the FASTA parser and through its own position scanner at the same time. As a result:

• the reported positions are never correct;
• sequences get corrupted once a file is larger than the parser's read buffer. With the current code, a 5 KB file of 50 sequences yields only 9 correct sequences.

The comparative assembler uses it through DeltaAlignmentCollection. I'd like to fix this separately.