[Proposal] Integrate AxoMEME as a hybrid MEME surrogate with fast MCP triage tool

[stevenweaver]

Summary

Integrate AxoMEME (a rapid deep-learning surrogate for MEME episodic-selection detection; veg/axomeme3) into Datamonkey3 as a hybrid feature with three surfaces sharing one inference core, plus a fast MCP tool for agent-driven triage.

AxoMEME predicts per-site MEME LRT statistics directly from an alignment + tree, without ML optimization. The value proposition is instant triage → confirm with a real MEME run, not replacing MEME.

Validation (why this is worth building)

Benchmarked against 188 real datamonkey MEME jobs (backup corpus, ground truth = each job's own .MEME.json):

Metric	Median	Mean
Spearman (pred LRT vs MEME LRT)	0.68	0.65
ROC AUC (calling MEME p≤0.05)	0.91	0.85
PR AUC	0.21	0.31

Accuracy generalizes to real, in-the-wild data. The rare-signal shape (high ROC, lower PR) means it's excellent at ranking selected sites but imprecise at the exact top — so it must be presented as triage, not a final caller.

Speed caveat (important): median server-side speedup vs MEME was only ~1.06× on CPU (grows with alignment size; IQR up to ~5×). The paper's ~1000× requires GPU and/or deep alignments. Design accordingly.

Architecture: one core, three surfaces

Surface	Runs where	Latency	Consumer
1. Client ONNX	Browser (ONNX Runtime Web)	seconds, zero infra	DM3 web app pre-screen
2. Server method	SLURM (GPU ideally)	seconds–min	large alignments
3. MCP tool	warm daemon (model preloaded)	sub-second	LLM agents / Claude

All three call one shared axomeme.predict(seqs, tree, max_species) core — different entry points, same logic. That refactor (extracting the logic from predict_regression_nexus_fasta.py's CLI main()) is the enabling step.

The key to "fast MCP": a warm daemon

Measured per-call cost is ~5s, but almost all of it is fixed startup (torch import + checkpoint load); the actual matrix math is <1s. Therefore:

Do NOT shell out to the CLI per MCP call — that pays the startup tax every time. Run a long-lived worker that loads the model once and serves inference over a socket. MCP call → warm worker → sub-100ms typical.

MCP tool contract

Add axomeme_scan as a tool on the existing DM3 MCP server (mcp.datamonkey.org/mcp) — reuse its auth; no new server.

{
  "name": "axomeme_scan",
  "description": "Rapid DL scan for episodic diversifying selection (MEME surrogate). Per-site predicted LRT + calls in <1s. Triage before a full MEME run.",
  "input_schema": {
    "alignment": "FASTA or NEXUS inline, OR a datamonkey job id",
    "tree":      "Newick (optional; uses embedded/estimated if omitted)",
    "threshold": "percentile for Tier-1 calls (default 98)"
  }
}
// returns: { n_sites, sites:[{site,pred_lrt,call,aa}], n_selected, elapsed_ms,
//            note:"surrogate — confirm hotspots with a full MEME run" }

The job id input is the hybrid payoff: an agent can chain pull job → scan → if hotspots, submit real MEME.

Example agent workflow

1. User: "Is there positive selection in this HIV env alignment worth a full analysis?"
2. Agent → axomeme_scan → 84ms → "8 Tier-1 sites: 140, 160, 275…"
3. Agent: "Sites 140 & 160 are known antigenic — worth confirming. Launch the authoritative MEME?"
4. User: "yes" → Agent → existing submit_meme MCP tool → job id → minutes later, rigorous result.
5. Agent reconciles: "MEME confirmed 6 of 8; AxoMEME over-called 2."

AxoMEME MCP = instant triage; DM3 MCP = authoritative confirmation; chained by the agent.

Build plan (phased, each independently shippable)

• Phase 0 — Refactor to library. Extract axomeme.predict() from the CLI. Enables everything. (~0.5d)
• Phase 1 — Warm daemon. Wrap predict() in a preloaded socket server; benchmark sub-second warm latency. (the "fast" in fast MCP)
• Phase 2 — MCP tool. Add axomeme_scan handler forwarding to the daemon on the existing DM3 MCP server; add job_id chaining.
• Phase 3 — Client ONNX pre-screen in the web app (Manhattan plot before a MEME submit).
• Phase 4 — SLURM method (GPU) for large alignments, sharing the core; render in the existing MEME result viewer (shared LRT/site schema).

Caveats to bake in

• GPU for real speed — daemon should sit on a GPU node; CPU is fine only for small alignments.
• Surrogate ≠ truth — every response carries the "confirm with MEME" note; triage-then-confirm, never replacement.
• Large-alignment tail — a 2084-site job took ~28 min on CPU. Daemon needs a request timeout + size cap that falls back to "submit a SLURM job instead."

Open questions (need DM3-backend context)

• DM3 MCP server's tool-registration mechanism (where/how to add axomeme_scan) — likely in veg/service-datamonkey.
• Is a GPU partition reachable from the DM3 backend for the warm daemon?
• Reuse of the MEME result renderer for AxoMEME output.

---

Validation artifacts (188-job pilot: metrics + speedup, SLURM harness) available on silverback under …/axomeme/validation/datamonkey-pilot/.

[stevenweaver]

⚠️ RETRACTED — see correction below. This comment assumed a server-side MEME queue that DM3 does not have (analyses run client-side via WASM HyPhy). The capacity/deflection/prioritization projections are invalid; disregard them. Kept for history.

---

Projected impact on MEME upload/capacity limits

Framing first (important): AxoMEME does not reduce the cost of any given MEME run — if a user still submits MEME, it costs the same. So this is not a raw-speedup argument. The mechanism that moves upload limits is triage-driven demand reduction: a free client-side/MCP pre-screen lets users avoid submitting MEME jobs that won't yield anything, lowering effective queue/CPU load so the nominal cap can be raised.

All numbers below are projections with stated assumptions, grounded in the 188-job real-datamonkey pilot.

What the pilot data shows

• 30% of MEME jobs found 0 significant sites (p ≤ 0.05) — skippable if the user saw a preview first.
• MEME cost is heavily right-skewed: median 42s, p90 323s, max 11,669s (~3.2 hr). Upload limits are set by this expensive tail, not the median.
• AxoMEME triage accuracy on these jobs: ROC AUC 0.91 / Spearman 0.68 — reliable enough to flag "nothing here, don't bother" vs "worth a full run."

Projected throughput headroom

If a client-side/MCP AxoMEME preview deflects a fraction of the no-signal submissions:

Scenario	Assumption	Submissions deflected	Effective throughput headroom
Conservative	catches 40% of no-signal jobs	~12%	~1.14×
Moderate	catches 60%	~18%	~1.22×
Optimistic	catches 80%	~24%	~1.32×

So a realistic read is a ~15–30% increase in effective MEME capacity → the upload/size cap could be raised by a comparable factor without adding hardware, purely from deflected demand. (Headroom = 1/(1−deflected_fraction), assuming deflected jobs carry ~average cost — conservative, since no-signal jobs skew toward the cheap end so the count relief may exceed the cost relief.)

Additional, harder-to-quantify upside

• Tail deflection. If even a handful of the largest jobs (the 3+ hour outliers) are pre-screened away, the peak load that sets the cap drops disproportionately — this could justify raising the per-job size limit more than the throughput math alone suggests.
• GPU server method (Phase 4). For users who do want a full scan on a huge alignment, a GPU-backed AxoMEME "full scan" offers a fast approximate result at a fraction of MEME's CPU cost — a separate, larger lever on the size cap for the approximate-answer use case.

Caveats

• These are demand-side projections, dependent on adoption (users actually running the preview before submitting) and on the deflection assumptions above — not a guaranteed compute saving.
• On CPU, AxoMEME's own per-job cost ≈ MEME's, so the savings come entirely from avoided MEME submissions, not from replacing MEME compute.
• Recommend validating the deflection rate with a short A/B: show the preview to a fraction of users and measure the actual drop in no-signal MEME submissions before committing to a specific new limit.

Model + per-job data: …/axomeme/validation/datamonkey-pilot/results/metrics.csv on silverback.

[stevenweaver]

⚠️ Correction — retracting the capacity/limits framing above

The projections in my earlier comment assumed a server-side MEME queue whose load AxoMEME could relieve by deflecting or prioritizing jobs. After reviewing this repo, that model is wrong and those numbers should be disregarded:

• DM3 runs analyses client-side via WebAssembly HyPhy (WasmAnalysisRunner), or as a user-owned backend job (BackendAnalysisRunner). There is no shared MEME queue to relieve.
• Therefore "deflect no-signal jobs to free capacity" is meaningless, and queue prioritization would be both irrelevant and unfair to other users. Retracted.
• The MEME limit is maxMEMESize = maxFELSize = **500 sequences** (per HyPhyGlobals.ibf); the 5,000/12,000 figures are the file upload ceiling, not MEME's cap. The cap exists because MEME's likelihood optimization is heavy — especially in-browser.

The real question isn't "how much capacity does it save" — it's "what would it take to make AxoMEME runnable and displayable in DM3 at all." Analysis below.

Integration reality: DM3 is a HyPhy pipeline end-to-end

Every method in DM3 is a HyPhy analysis:

1. declared in src/lib/config/methodOptions.toml,
2. executed as a HyPhy .bf batch file (WASM image) or backend Socket.IO job,
3. producing a standard HyPhy result JSON consumed by AnalysisResultViewer → MemeVisualization, plus IndexedDB storage and batch export.

AxoMEME is a PyTorch/ONNX model and fits none of this:

• it is not a .bf, so neither existing runner can execute it;
• its output is a flat CSV, not a HyPhy MEME JSON, so the viewer/storage/export pipeline can't consume it.

What actually needs to happen (dependency order)

1. A non-HyPhy runner. Both current runners are HyPhy. AxoMEME needs a third execution path plugged into the BaseAnalysisRunner abstraction — ONNX-in-browser (onnxruntime-web) or a backend inference endpoint. Net-new; nothing non-HyPhy has plugged in before.
2. Output conforming to the HyPhy MEME JSON schema (MLE.headers / MLE.content, per-site LRT + p-value columns). Highest leverage, smallest task: if AxoMEME emits this, MemeVisualization, storage, and export work for free. Today it emits a bespoke CSV.
3. Honest result semantics. AxoMEME produces a predicted LRT with no calibrated p-value. Reusing the MEME schema risks the UI presenting fabricated p-values. Needs either (a) calibration: map predicted LRT → empirical p-value using the 188-job pilot, or (b) a distinct "approximate scan" result view labeled as such.

Bottom line

The model's accuracy is validated (site-level Neutral calls 99% reliable; ROC AUC 0.91; Spearman 0.68 vs real MEME). The blocker to usefulness in DM3 is not accuracy or capacity — it's that AxoMEME is a foreign artifact to a HyPhy-only pipeline. The concrete, minimal path to "useful" is: (2) make it emit MEME-shaped JSON, then (1) give it a runner, with (3) honest p-value semantics so we don't misrepresent a surrogate as an exact test.

[stevenweaver]

The actual goal: AxoMEME as a client-side pre-screen (not a HyPhy method)

Reframing per discussion. AxoMEME does not need to be a HyPhy method, emit MEME JSON, or plug into the result-visualization pipeline. It is a fast, client-side pre-screen shown before the user commits to a real MEME/FEL WASM run — advisory only.

Why this fits DM3 cleanly

There is already a precedent for "pre-run advice": SequenceWarnings.svelte takes file metrics and renders advisory cards ("short alignment", "duplicate sequences", "small sample size") before analysis, with no dependency on the HyPhy result pipeline.

AxoMEME pre-screening is the same pattern, one level richer: take the loaded alignment, run the ONNX model in-browser, and render an advisory like:

• "Predicted episodic selection at ~8 sites (positions 140, 160, 275, …). A full MEME run is likely worthwhile." or
• "No strong selection signal detected across N sites — MEME may return few/no hits."

It runs in the DataTab → Analyze transition (AnalyzeTab.svelte already dispatches executionMode), reuses the alignment/tree the user already loaded, and adds zero load elsewhere.

What's needed (small, self-contained)

1. Client ONNX inference — load MEME_transformer.onnx (already exists in veg/axomeme3) via onnxruntime-web; feed the parsed alignment + tree. No new backend, no HyPhy, no .bf.
2. A pre-screen advisory component modeled on SequenceWarnings.svelte — consumes AxoMEME's per-site output, renders a summary card + optional mini Manhattan sparkline, shown before the user clicks Run.
3. Advisory-only framing — it's a heads-up, explicitly labeled "approximate / deep-learning surrogate — confirm with MEME." No p-values, no result storage, no export entanglement. Sidesteps the calibration problem entirely because we never claim significance, just "worth a look / probably not."

Why this is the right scope

• No fairness/queue/capacity concerns (those were retracted above) — it's per-user, client-side, advisory.
• No pipeline surgery — it does not masquerade as a HyPhy result, so MemeVisualization/storage/export are untouched.
• Plays to the model's proven strength: site-level Neutral calls are 99% reliable and ROC AUC is 0.91 — exactly the "is there likely anything here, and roughly where" question a pre-screen answers.

Suggested next step

Spike #1: wire onnxruntime-web + MEME_transformer.onnx into a minimal DM3 branch and confirm in-browser inference produces the per-site scan on a demo alignment. That de-risks the only novel piece (ONNX in the DM3 build); the advisory component is then straightforward.

[stevenweaver]

Model shortcomings (be clear-eyed before building the pre-screen)

Grouped by strength of evidence. Measured numbers are from the 188-job real-datamonkey pilot (metrics.csv).

Measured in the pilot (hard evidence)

1. Accuracy degrades on deep alignments — exactly where the pre-screen is most wanted.
Median Spearman vs MEME LRT falls monotonically with sequence depth:

depth	n	median Spearman
<30 seqs	92	0.688
30–100	59	0.677
100–300	23	0.646
300+	8	0.479

The motivating use case was scanning alignments too big for in-browser MEME — but that's where AxoMEME is least reliable. Almost certainly the max_species=256 Faith's-PD downsampling discarding signal on deep trees.

2. It fails — sometimes anticorrelates — on shallow alignments.
Worst performers cluster at 3–19 sequences (Spearman −0.14, 0.00, 0.02). Few taxa → little phylogenetic signal for the row-attention. A "no signal" advisory on a 3-sequence alignment is near-worthless and occasionally backwards.

3. Poor precision at the very top. Per-site PPV ~24%; top-10 often 2/10. Ranks well (ROC AUC 0.91) but its single most-confident calls are frequently wrong. Good for "roughly where," bad for "these exact sites."

Structural (inherent to the design)

4. Trained only on mammals (VGP/TOGA). Every training alignment is a mammalian gene; datamonkey users submit viruses/bacteria/plants. The pilot held up on average, but Hannah's Dengue test showed a collapse on fast-evolving RNA viruses (negative Spearman). Performance is organism-dependent in ways a mammalian-trained model can't fully cover.

5. It can't beat its teacher. It approximates HyPhy MEME's LRT under MEME's own assumptions. Where MEME is misspecified (recombination, alternative genetic codes/mtDNA, extreme rate variation), AxoMEME inherits those failures plus its own approximation error.

6. No calibrated uncertainty. Emits a predicted LRT, not a p-value, with no notion of "unsure here." This is the fundamental reason it can only ever be advisory, never a significance test.

7. Needs a tree with branch lengths. The patristic-distance attention bias requires a real tree; when absent, the pipeline falls back to HyPhy branch-length estimation — reintroducing the slow step a surrogate is meant to avoid.

Still unverified (flagging honestly)

• Organism breakdown of the pilot — aggregate metrics may hide a bimodal "great on mammal-like, poor on viral" split. Not yet checked.
• Whether the deep-alignment drop is fully explained by max_species (testable by re-running deep jobs at higher caps).
• Whether predicted-LRT → empirical-p calibration is consistent across organism types (if not, it undermines even advisory use).

Implication for the pre-screen

The two that matter most are #1 and #2: weakest on the deep alignments that motivated the feature, and potentially misleading on shallow ones. The advisory should therefore be gated to the validated range (~30–300 sequences) with an explicit "not reliable for very small or very large alignments," rather than a blanket "no signal detected." Where it's outside that range or organism is out-of-distribution (e.g. viral), it should say so rather than render a confident-looking scan.

[stevenweaver]

Progress: surface 1 of 3 is shipped.

The client-side ONNX path (AxoMEME running entirely in-browser via onnxruntime-web, no server round-trip) landed in #153 and is live behind a Beta tag, with results rendered by the AxomemeVisualization component in hyphy-scope.

Per the validation in this issue, it is presented as a site ranker, not a caller: the default calling mode is percentile-based, and the UI suppresses controls the model cannot honour.

Still open from this proposal: the server-side surface, the MCP triage tool, and the hybrid triage-then-confirm handoff to a real MEME run.