Development

CHANGELOG.md

@@ -3,6 +3,54 @@
 All notable changes to FunVIP are documented here. This project adheres to
 [Semantic Versioning](https://semver.org/).
 
+## [1.0.2] - 2026-09-30
+
+Faster outgroup selection and tree drawing on large databases, earlier input
+validation, and a fix for runs at the default verbosity on Windows and macOS.
+Analysis results are unchanged.
+
+### Performance
+- **Outgroup selection groups the search table once.** Preparing the outgroup tasks
+  built a new `groupby` over the whole concatenated search table for every group,
+  which is O(groups x rows) and ran in the main process before the worker pool
+  started. On a 121,133-sequence, 5,266-genus ITS database this took about 2 h with no
+  log output. The table is now grouped once. At 1,500 groups and 300,000 search hits,
+  preparing the tasks went from 34.8 s to 0.15 s with the same task list.
+- **Single-gene groups interpret and draw each tree once.** With one gene, the
+  concatenated tree is a copy of the gene tree, but it was interpreted and rendered a
+  second time. When the tree, the trimmed alignment, the partition and the query and
+  outgroup lists of both datasets are identical, the concatenated tree now reuses the
+  gene tree's interpretation, SVG and report rows. Otherwise it is interpreted as
+  before. On the 121,133-sequence ITS database about half of the visualize step was
+  this duplicate work.
+
+### Fixed
+- **Crash on Windows and macOS at the default verbosity.** Outgroup selection and tree
+  interpretation handed the sequence records to their worker processes through a
+  module-level store that only fork-started workers inherit. Windows and macOS start
+  workers by spawning, so every run below `--verbose 3` stopped at the outgroup step
+  with `KeyError: 'list_FI'`. Each worker now receives the records when it starts.
+  The bundled test presets use `--verbose 3`, which is why `--test` runs passed.
+- **Free text in a gene column no longer reaches GenMine.** A cell was treated as an
+  accession whenever an accession appeared anywhere in it, and the whole cell was
+  sent to GenMine, which failed on free text and accession lists. A cell is now
+  downloaded only when it is a single token. Other cells containing an accession stop
+  the run at input validation, reporting the table, line and column. FASTA-formatted
+  cells (starting with `>`) are read as sequences even when the header holds an
+  accession.
+- **IDs that collide after non-ASCII replacement stop at input validation.**
+  `LÖ21-04` and `LO21-04`, or IDs that differ only in a Unicode hyphen, became the
+  same ID and were merged into one record, or failed later as a colliding genus or
+  datatype. Each such pair is now reported with both original strings, across DB and
+  query tables. An ID repeated verbatim is still merged with a warning, as before.
+- IDs without any letter or digit (`--`, `...`, a lone dash) are rejected as empty
+  IDs.
+
+### Documentation
+- The `--outgroupoffset` help and `docs/parameters.md` now state that the value also
+  discards every search hit at or below it before clustering, besides setting the
+  bitscore gap between ingroup and outgroup.
+
 ## [1.0.1] - 2026-09-10
 
 ### Fixed
@@ -82,5 +130,6 @@ so rebuild the conda environment instead of `pip install --upgrade`. See
   the fitted regression line.
 - Removed confirmed-dead / broken code paths.
 
+[1.0.2]: https://github.com/Changwanseo/FunVIP/releases/tag/v1.0.2
 [1.0.1]: https://github.com/Changwanseo/FunVIP/releases/tag/v1.0.1
 [1.0.0]: https://github.com/Changwanseo/FunVIP/releases/tag/v1.0.0

docs/parameters.md

@@ -89,7 +89,7 @@ do not edit by hand. Regenerate after changing the CLI._
 | `--allow-innertrimming` | flag | Turn off FunVIP adjustment to not to trim inner alignment columns, default: False |
 | `--criterion` | str | Model-selection criterion for modeltest, one of [AIC, AICc, BIC]. default : BIC |
 | `--noavx` | flag | Do not use AVX for RAxML, default: False |
-| `--outgroupoffset` | int | outgroupoffset value. Highering this value may select more distant outgroup, default : 20 |
+| `--outgroupoffset` | int | Bitscore gap between ingroup and outgroup. Outgroup candidates must score more than this below the weakest ingroup hit, so a higher value selects a more distant outgroup. Search hits with a bitscore at or below this value are also discarded before clustering. default : 20 |
 | `--nosuspicious` | flag | Do not include suspicious samples for sequence-set. Mostly for metabarcoding analysis. May deduce inaccurate result with problematic database. |
 | `--terminate` | flag | Terminate FunVIP run when critical error detected |
 

funvip/src/cluster.py

@@ -205,11 +205,16 @@ def cluster(FI, df_search, opt):
 
 
 ### Append outgroup to given group-gene dataset by search matrix
-# V.list_FI shared with outgroup-append pool workers via fork copy-on-write, set
-# before the Pool is created, instead of being pickled into every task tuple.
+# V.list_FI shared with outgroup-append pool workers once per worker (inherited
+# through fork on Linux, sent to each spawned worker on Windows and macOS) instead
+# of being pickled into every task tuple.
 _OUTGROUP_SHARED = {}
 
 
+def _share_outgroup_data(list_FI):
+    _OUTGROUP_SHARED["list_FI"] = list_FI
+
+
 def append_outgroup(df_search, gene, group, path, opt):
     logging.info(f"Appending outgroup on group: {group}, Gene: {gene}")
     list_FI = _OUTGROUP_SHARED["list_FI"]
@@ -433,11 +438,13 @@ def outgroup_append_opt_generator(V, path, opt):
 
     # if concatenated analysis is true
     # concatenated
+    df_group = None
     for group in V.dict_dataset:
         if "concatenated" in V.dict_dataset[group]:
             try:
-                df = V.cSR
-                df_group = df.groupby(df["query_group"])
+                if df_group is None:
+                    df = V.cSR
+                    df_group = df.groupby(df["query_group"])
                 df_group_ = df_group.get_group(group)
                 # Generating outgroup opt for multiprocessing
                 for gene in V.dict_dataset[group]:
@@ -538,13 +545,15 @@ def pipe_cluster(V, opt, path):
 def pipe_append_outgroup(V, path, opt):
     opt_append_outgroup = outgroup_append_opt_generator(V, path, opt)
 
-    # Share V.list_FI with pool workers via fork copy-on-write (set before the Pool)
-    # instead of pickling the whole list into every task.
-    _OUTGROUP_SHARED["list_FI"] = V.list_FI
+    # Share V.list_FI with pool workers once per worker instead of pickling the
+    # whole list into every task.
+    _share_outgroup_data(V.list_FI)
 
     # run multiprocessing start
     if opt.verbose < 3:
-        p = mp.Pool(opt.thread)
+        p = mp.Pool(
+            opt.thread, initializer=_share_outgroup_data, initargs=(V.list_FI,)
+        )
         result_append_outgroup = p.starmap(append_outgroup, opt_append_outgroup)
         p.close()
         p.join()

funvip/src/command.py

@@ -315,7 +315,7 @@ def get_args(self) -> argparse.Namespace:
         )
         group_advanced.add_argument(
             "--outgroupoffset",
-            help="outgroupoffset value. Highering this value may select more distant outgroup, default : 20",
+            help="Bitscore gap between ingroup and outgroup. Outgroup candidates must score more than this below the weakest ingroup hit, so a higher value selects a more distant outgroup. Search hits with a bitscore at or below this value are also discarded before clustering. default : 20",
             type=int,
         )
         group_advanced.add_argument(

funvip/src/tree_interpretation_pipe.py

@@ -7,11 +7,13 @@
 from funvip.src.reporter import Singlereport
 from funvip.src.exceptions import TreeError
 import traceback
-from copy import deepcopy
+from copy import copy, deepcopy
+from Bio import SeqIO
 import pandas as pd
 import re
 import sys
 import os
+import filecmp
 import shutil
 import psutil
 import logging
@@ -22,11 +24,17 @@
 ### For single dataset
 # Input : out, group, gene, V, path, opt
 # Whole-run FI collections (V.dict_hash_FI / V.list_FI) shared with
-# interpretation-pool workers via fork copy-on-write, set before the Pool is
-# created, instead of being pickled into every (group, gene) task tuple.
+# interpretation-pool workers once per worker (inherited through fork on Linux,
+# sent to each spawned worker on Windows and macOS) instead of being pickled into
+# every (group, gene) task tuple.
 _INTERP_SHARED = {}
 
 
+def _share_interpretation_data(funinfo_dict, funinfo_list):
+    _INTERP_SHARED["funinfo_dict"] = funinfo_dict
+    _INTERP_SHARED["funinfo_list"] = funinfo_list
+
+
 def pipe_module_tree_interpretation(
     out,
     group,
@@ -41,7 +49,7 @@ def pipe_module_tree_interpretation(
 ):
     # time_start = time()
 
-    # Read the whole-run FI collections from the fork-inherited shared store rather
+    # Read the whole-run FI collections from the per-worker shared store rather
     # than receiving a freshly pickled copy of the entire universe per task.
     funinfo_dict = _INTERP_SHARED["funinfo_dict"]
     funinfo_list = _INTERP_SHARED["funinfo_list"]
@@ -647,6 +655,76 @@ def _safe_pipe_module_tree_visualization(*args):
         return None
 
 
+def _single_gene_twin(V, path, opt, group):
+    genes = [g for g in V.dict_dataset[group] if g != "concatenated"]
+    if len(genes) != 1 or "concatenated" not in V.dict_dataset[group]:
+        return None
+    gene = genes[0]
+
+    dataset_gene = V.dict_dataset[group][gene]
+    dataset_concat = V.dict_dataset[group]["concatenated"]
+    for attr in ("list_qr_FI", "list_og_FI"):
+        if [FI.hash for FI in getattr(dataset_gene, attr)] != [
+            FI.hash for FI in getattr(dataset_concat, attr)
+        ]:
+            return None
+
+    def tree_file(g):
+        return f"{path.out_tree}/hash/hash_{opt.runname}_{group}_{g}.nwk"
+
+    def alignment_file(g):
+        return f"{path.out_alignment}/hash/{opt.runname}_hash_trimmed_{group}_{g}.fasta"
+
+    for get_file in (tree_file, alignment_file):
+        a, b = get_file(gene), get_file("concatenated")
+        if not (
+            os.path.isfile(a)
+            and os.path.isfile(b)
+            and filecmp.cmp(a, b, shallow=False)
+        ):
+            return None
+
+    partition = V.partition.get(group)
+    if partition is None or partition.get("order") != [gene]:
+        return None
+    try:
+        width = len(next(SeqIO.parse(alignment_file(gene), "fasta")).seq)
+    except StopIteration:
+        return None
+    if partition.get("len", {}).get(gene) != width:
+        return None
+
+    return gene
+
+
+def _twin_interpretation(tree_info, path, opt):
+    group, gene = tree_info.group, tree_info.gene
+    out = f"{path.out_tree}/{opt.runname}_{group}"
+    for prefix in (f"{path.out_tree}/hash_{opt.runname}_{group}", out):
+        shutil.copy(
+            f"{prefix}_{gene}_original.svg", f"{prefix}_concatenated_original.svg"
+        )
+    shutil.move(f"{out}_concatenated.nwk", f"{out}_concatenated_original.nwk")
+    shutil.copy(f"{out}_{gene}.nwk", f"{out}_concatenated.nwk")
+
+    twin = copy(tree_info)
+    twin.gene = "concatenated"
+    twin.tree_name = f"{path.out_tree}/hash/hash_{opt.runname}_{group}_concatenated.nwk"
+    return twin
+
+
+def _twin_visualization(tree_info, report_list, path, opt):
+    out = f"{path.out_tree}/{opt.runname}_{tree_info.group}"
+    shutil.copy(f"{out}_{tree_info.gene}.svg", f"{out}_concatenated.svg")
+
+    twin_report_list = []
+    for report in report_list:
+        twin_report = copy(report)
+        twin_report.update_gene("concatenated")
+        twin_report_list.append(twin_report)
+    return twin_report_list
+
+
 def pipe_tree_interpretation(V, path, opt):
     # Generate tree_interpretation opt to run
     # tree_interpretation_opt = []
@@ -661,11 +739,9 @@ def pipe_tree_interpretation(V, path, opt):
     funinfo_list = V.list_FI
     hash_dict = V.dict_hash_name
 
-    # Share the whole-run FI collections with pool workers via fork copy-on-write
-    # (must be set before the Pool below is created) instead of pickling them into
-    # every task; workers read them from _INTERP_SHARED.
-    _INTERP_SHARED["funinfo_dict"] = funinfo_dict
-    _INTERP_SHARED["funinfo_list"] = funinfo_list
+    # Share the whole-run FI collections with pool workers once per worker instead
+    # of pickling them into every task; workers read them from _INTERP_SHARED.
+    _share_interpretation_data(funinfo_dict, funinfo_list)
 
     # Generate options using generator
     def generate_interpretation_opt():
@@ -720,23 +796,59 @@ def generate_interpretation_opt():
                             f"Failed interpreting tree {group} {gene} because no outgroup available"
                         )
 
-    tree_interpretation_opt = generate_interpretation_opt()
-
-    tree_info_list = []
+    tree_interpretation_opt = list(generate_interpretation_opt())
+
+    task_keys = [(option[1], option[2]) for option in tree_interpretation_opt]
+    task_key_set = set(task_keys)
+    twin_gene = {}
+    for group in V.dict_dataset:
+        gene = _single_gene_twin(V, path, opt, group)
+        if (
+            gene is not None
+            and (group, gene) in task_key_set
+            and (group, "concatenated") in task_key_set
+        ):
+            twin_gene[group] = gene
+    if twin_gene:
+        logging.info(
+            f"Reusing single gene tree interpretation for the concatenated tree of {len(twin_gene)} groups"
+        )
+    run_opt = [
+        option
+        for option in tree_interpretation_opt
+        if not (option[1] in twin_gene and option[2] == "concatenated")
+    ]
 
     ## Tree interpretation - outgroup, reconstruction(solve_flat), collapsing
     if opt.verbose < 3:
-        with mp.Pool(opt.thread) as p:
-            tree_info_list.extend(
-                p.starmap(_safe_pipe_module_tree_interpretation, tree_interpretation_opt)
-            )
+        with mp.Pool(
+            opt.thread,
+            initializer=_share_interpretation_data,
+            initargs=(funinfo_dict, funinfo_list),
+        ) as p:
+            run_result = p.starmap(_safe_pipe_module_tree_interpretation, run_opt)
 
     else:
         # non-multithreading mode for debugging
-        tree_info_list = [
-            pipe_module_tree_interpretation(*option)
-            for option in tree_interpretation_opt
-        ]
+        run_result = [pipe_module_tree_interpretation(*option) for option in run_opt]
+
+    interpreted = {
+        (option[1], option[2]): tree_info
+        for option, tree_info in zip(run_opt, run_result)
+    }
+    twin_of = {}
+    tree_info_list = []
+    for group, gene in task_keys:
+        if group in twin_gene and gene == "concatenated":
+            source = interpreted[(group, twin_gene[group])]
+            if source is None:
+                tree_info_list.append(None)
+                continue
+            twin = _twin_interpretation(source, path, opt)
+            twin_of[id(twin)] = source
+            tree_info_list.append(twin)
+        else:
+            tree_info_list.append(interpreted[(group, gene)])
 
     # Drop genera that failed interpretation (guarded above) so one bad tree
     # does not sink the whole run; failures are logged as [TREE INTERPRETATION FAILED].
@@ -761,25 +873,44 @@ def generate_interpretation_opt():
         V_dict_hash_FI[key] = (FI.original_id, FI.hash, FI.adjusted_group)
 
     # Generate options using generator
+    visualization_list = [ti for ti in tree_info_list if id(ti) not in twin_of]
+
     def generate_visualization_opt():
-        for tree_info in tree_info_list:
+        for tree_info in visualization_list:
             yield (tree_info, V.tup_genus, V_dict_hash_FI, path, opt)
 
     # Generate visualization option to run
     tree_visualization_opt = generate_visualization_opt()
     ## Tree visualization
     if opt.verbose < 3:
         with mp.Pool(opt.thread) as p:
-            tree_visualization_result = p.starmap(
+            visualization_result = p.starmap(
                 _safe_pipe_module_tree_visualization, tree_visualization_opt
             )
 
     else:
         # non-multithreading mode for debugging
-        tree_visualization_result = [
+        visualization_result = [
             pipe_module_tree_visualization(*option) for option in tree_visualization_opt
         ]
 
+    visualized = {
+        id(tree_info): report_list
+        for tree_info, report_list in zip(visualization_list, visualization_result)
+    }
+    tree_visualization_result = []
+    for tree_info in tree_info_list:
+        if id(tree_info) not in twin_of:
+            tree_visualization_result.append(visualized[id(tree_info)])
+        else:
+            source = twin_of[id(tree_info)]
+            if visualized[id(source)] is None:
+                tree_visualization_result.append(None)
+            else:
+                tree_visualization_result.append(
+                    _twin_visualization(source, visualized[id(source)], path, opt)
+                )
+
     # Drop genera that failed visualization (guarded) before flattening
     tree_visualization_result = [r for r in tree_visualization_result if r is not None]