Skip to content

Add single-end long-read support behind a hidden --platform option - #30

Merged
pdimens merged 3 commits into
mainfrom
claude/long-read-support
Oct 16, 2026
Merged

pdimens merged 3 commits into
mainfrom
claude/long-read-support

Conversation

@pdimens

@pdimens pdimens commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a hidden --platform option to arachne align (short, pacbio, ont) that selects the minibwa preset and, for the long-read platforms, switches the pipeline to single-end reads. The default is short, which keeps the existing short-read behaviour. The flag is not shown in --help and is not documented in docs/usage.md.

With --platform pacbio or --platform ont, align takes one FASTQ (REF.fa R1.fq) that is already sorted by barcode, and each read is mapped on its own with minibwa's lr preset. minibwa has no PacBio- or ONT-specific presets, so both use lr for now; they are separate entries in one table (platforms in aligner/mapping.go) so they can diverge later.

What changed

Option and plumbing

  • --platform (hidden, default short), validated in the CLI. The argument count follows the platform: three for short, two for the long-read platforms.
  • Platform profile (minibwa preset, @RG PL value, single-end or not) and LookupPlatform.
  • @RG PL is now ILLUMINA, PACBIO or ONT instead of always ILLUMINA.

Reader

  • OpenFastQ reads a single FASTQ. A trailing /1 is trimmed from the read name only if present, because long-read names (UUIDs, movie/zmw/ccs) would otherwise lose two characters.

Alignment

  • Alignments carry single_end. The empty second slot of each read stays as a placeholder so the pair-oriented code is untouched, but it is never written to the output.
  • Single-end reads get no improper-pair penalty, and IsUnmapped only checks the position (the short-read score rule would demote them).
  • Candidate selection for single-end reads keeps only hits that cover mostly the same part of the read as the best hit (overlap of at least 0.5 of the shorter span). The other part of a chimeric read stays a split piece. Without this filter the MAPQ log-loss on simulated data roughly tripled; any threshold from 0.1 to 0.9 gave identical results.
  • MAPQ estimation measures scores from the top score when they fall below -300 (log10). Over a long read, exp() of the score underflowed to 0, the normalisation became 0/0, and MAPQ came out as NaN cast to an integer (-9223372036854775808). The shift is applied only in that range, so ordinary scores are untouched.
  • The EM's alignment-only MAPQ ceiling is not applied to single-end reads. It assumes few mismatches and would pin every long read at MAPQ 0.

Output

  • Single-end reads are written as unpaired records: no pair flags, mate fields unset.
  • The duplicate flag (0x400) is now set outside the paired-record block so it is written for unpaired records too. Paired output is unchanged.
  • Split pieces of a long read are flagged supplementary (0x800); short reads keep 0x100.

Testing

  • go vet, gofmt and go test ./... pass.
  • Short-read output is identical to current main (sorted records compared, EM and --rfa) on the simulated paired set I used throughout.
  • New tests cover: platform lookup and preset validity, scoring and IsUnmapped for single-end reads, MAPQ normalisation with extreme scores (it fails with the fix disabled), the EM ceiling, single-end record flags, GetAlignments in single-end mode (unique, repeated and chimeric reads), candidate competition, the single-FASTQ reader, and the CLI argument counts.
  • Simulated long reads (HiFi-like 0.5% error, ONT-like 2%, 5% and 8%) with dispersed, tandem and partial repeats, chimeric reads, and lone-read barcodes: all runs finish cleanly, strand and @RG are correct, every chimeric read gets a primary plus a split record, exact duplicates are flagged, and --rfa works. With identical repeat copies about 35% of reads are misplaced, nearly all at MAPQ 1 to 9, so calibration behaves as expected.

Tuning

I swept the new long-read settings before settling on them (three 4,000-read datasets, then a hold-out of three 6,000-read datasets with new seeds). The window scaling, a per-platform EM error rate and a long-read indel rate had no measurable benefit, so they are not in this PR. On the hold-out the final settings were as good or better than the earlier ones (83 vs 85 misplaced reads, 4 vs 5 wrong at MAPQ >= 10).

Limitations

  • Simulated data only. The error model (substitutions, insertions and deletions at 40:30:30) and the repeat structure are my assumptions, and real PacBio and ONT reads have systematic errors the simulation lacks. Please check the long-read MAPQ values on real data before relying on them.
  • Narrow candidate sets. Arachne can only weigh alternatives that minibwa reports. With the lr preset minibwa reported a second repeat copy at some small differences and not at others, depending on seed layout, so long-read candidate sets are narrower than short-read ones.
  • One split record per read. A read with several chimeric pieces gets only the best non-overlapping one.
  • arachne prep is still paired-only, so long-read FASTQ must already be sorted by barcode.
  • PacBio and ONT currently share the same preset.

The simulators and scoring scripts used for the sweeps are not part of this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BUHUTLHfLoVtWa4XbCL9ZH


Generated by Claude Code

--platform takes short (default), pacbio or ont and selects the minibwa
preset: short -> sr, pacbio and ont -> lr (minibwa has no platform-specific
presets yet). The flag is hidden from --help and defaults to short, so
the existing short-read behavior is unchanged: alignment output matches the
previous build byte for byte after sorting.

Only short is exercised end to end; the pipeline still reads paired FASTQ.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BUHUTLHfLoVtWa4XbCL9ZH
With the hidden --platform option set to pacbio or ont, align takes one
FASTQ (REF.fa R1.fq) and maps each read on its own with minibwa's lr preset.
The default short platform is unchanged: short-read output is identical to
before for both EM and RFA.

Reader
- OpenFastQ reads one FASTQ; a trailing /1 is trimmed from the read name only
  if present, since long read names rarely have one.

Alignment
- Alignments carry single_end. The empty second slot of each read stays as a
  placeholder so the paired bookkeeping is untouched, but it is not written.
- No improper pair penalty for single-end reads; IsUnmapped only checks the
  position (the short-read score rule would wrongly demote them).
- Candidate window grows with the best score (10%, floor 17) and only hits
  covering mostly the same part of the read compete; the other part of a
  chimeric read stays a split piece. Both fractions are untuned.
- MAPQ estimation measures scores from the top score when they fall below
  -300 (log10). Over a long read exp() underflowed and MAPQ became NaN.
- The EM's alignment-only MAPQ ceiling is not applied to single-end reads:
  it assumes few mismatches and would pin every long read at MAPQ 0.
- Per-platform EM error rate (short 0.001, pacbio 0.001, ont 0.02; untuned
  starting values) and indel rate equal to the error rate for long reads.
  --em-error-rate still overrides.

Output
- Unpaired records: no pair flags, mate fields unset, duplicate flag kept.
- Split pieces of a long read are flagged supplementary (0x800).
- @rg PL is ILLUMINA, PACBIO or ONT.

arachne prep still takes paired FASTQ; long reads must already be sorted by
barcode.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BUHUTLHfLoVtWa4XbCL9ZH
Swept the long-read settings on simulated data (HiFi-like 0.5% and ONT-like
5% and 8% error; dispersed, tandem and partial repeats; chimeric reads; 80%
lone-read barcodes; 3 datasets of 4000 reads, then a hold-out of 3 x 6000
with new seeds):

- candidate window fraction 0 to 0.1: identical; 0.2 and 0.4: slightly worse
  (more wrong reads at MAPQ >= 10). Removed; long reads use the same window
  as short reads.
- query-overlap filter: removing it roughly tripled MAPQ log-loss. Any
  threshold from 0.1 to 0.9 gave identical results, so 0.5 stays.
- EM error rate 0.001 to 0.01: identical; 0.02 to 0.1: one more misplaced
  read. The ONT-specific 0.02 had no support, so every platform uses 0.001
  (--em-error-rate still overrides).
- indel rate 0.1x to 3x the error rate: identical. Long reads keep the
  short-read indel rate.

On the hold-out the simplified settings were as good or better than the
previous ones (83 vs 85 misplaced, 4 vs 5 wrong at MAPQ >= 10, log-loss
0.0288 vs 0.0312). Short-read output is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BUHUTLHfLoVtWa4XbCL9ZH
@pdimens
pdimens merged commit 6a44190 into main Oct 16, 2026
1 check passed
@pdimens
pdimens deleted the claude/long-read-support branch October 16, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants