Paper: VariantValidator/Mutalyzer head-to-head on the injection corpus - #127
Merged
Conversation
…ding note - #112 Extend the injection-corpus comparison to the two sequence-aware validation services over their public REST APIs (measured 2026-08-18, VariantValidator 4.0.1, Mutalyzer; script caches and resumes, gives up if a service blocks sustained automated use and reports coverage). Weighted by the production error mix VariantValidator recovers 56.1% and Mutalyzer 36.2% vs 100% for clean_hgvs(); neither falsely corrects a valid input. Table S5 gains the two service columns and rejection notes. Discussion and Methods note that VariantValidator 4.0.0 embeds the LOVD syntax checker's suggestions in its API responses without applying them, so the local LOVD measurement also characterises that layer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Written by Claude
Extends the LOVD comparison (#125/#126) to the two sequence-aware validation services, measured over their public REST APIs on 2026-08-18 (VariantValidator 4.0.1, routed to its separate Ensembl endpoint for ENST; Mutalyzer normalize API). Same 3,419 injected cases, same gene-annotation-insensitive exact-match scoring.
Headline (weighted by the production error mix):
clean_hgvs()100% (by construction), VariantValidator 56.1%, Mutalyzer 36.2% (unweighted: 37.7% / 12.5%). Neither service falsely corrected a valid input. VariantValidator repairs whitespace, case, quotes, doubled colons and swapped gene/transcript at 94-99% per category, capped by 37/998 valid originals citing recent ENST accessions absent from its vvta_2025_02 database; Mutalyzer repairs whitespace and case only and rejects intronic positions on RefSeq transcript references (EINTRONIC, 112/998).Also adds the requested note, with citation: VariantValidator 4.0.0 (June 2026) embeds the LOVD HGVS syntax checker, and its API responses for unparseable input carry the checker's ranked suggestions (observed in 1,528/3,419 cases) without applying them, so the local LOVD measurement also characterises that layer of a current VariantValidator pipeline.
Operational notes: the fetch script caches to JSONL and resumes; VariantValidator's service tarpits sustained automated use, so the fetcher treats timeouts as throttle signals, backs off, and gives up after repeated dead batches, scoring VV over answered cases with coverage recorded in the facts (this run reached 100% coverage after a quiet period). Snakefile rule follows the new COMPUTED_FACTS pattern with a frozen-constants fallback; live rerun is opt-in via
--config vv_mutalyzer=true. Table S5 gains the two service columns; both docx targets build with all template values substituted.🤖 Generated with Claude Code
https://claude.ai/code/session_015662nbf1iezxdL9EMcckU7