Repository navigation
Development - #74
Merged
Merged
Development#74
Conversation
outgroup_append_opt_generator built a new groupby over the whole concatenated search table inside the loop over groups, so preparing the outgroup tasks cost O(groups x rows). It runs in the main process before the worker pool starts and logs nothing; on a 121,133-sequence, 5,266-genus ITS database it took about two hours while the outgroup selection itself took seven minutes. Build the groupby once, lazily inside the existing try so a missing query_group column still ends in the same per-group warning. On a synthetic table of 300,000 hits the task preparation went from 34.8 s to 0.15 s at 1,500 groups, with an identical task list. The three test datasets give byte-identical outputs to 1.0.1 before and after MAFFT. The new test counts groupby calls so a per-group regrouping cannot come back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZTJs2CAFcHw9Ye4k7p9Mx
With one gene, tree.py already copies the gene tree to the concatenated name, but tree interpretation and SVG rendering still ran for both, on identical trees. On a single-gene 121,133-sequence ITS database about half of the visualize step (1 h 14 min, 82.7 GB peak at 8 workers) was this duplicate work. For a group with exactly one gene, the concatenated tree now reuses the gene tree's interpretation: its output files are copied, the tree_info is aliased under the concatenated name, and the report rows are cloned with the gene set to "concatenated", all at the same positions as before so the merge into the result table sees the same order. The shortcut is taken only when the hash tree, the trimmed alignment, the partition and the query and outgroup lists of both datasets are identical; otherwise both are interpreted as before. On the sanghuangporus test (one ITS group) the tree is interpreted once and all 24 post-MAFFT outputs are byte-identical to 1.0.1; terrei and penicillium are multi-gene and unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZTJs2CAFcHw9Ye4k7p9Mx
A gene cell was treated as an accession whenever an accession pattern appeared anywhere in it, and the whole cell was sent to GenMine, which crashed on free text and accession lists hours into a large run. A cell is now sent to GenMine only when it is a single token; any other cell containing an accession is reported at input validation with its table, line and column. Cells in FASTA format are read as sequences even when the header holds an accession. Single accessions keep reaching GenMine as before, including formats the strict patterns miss (2 letters + 8 digits, RefSeq with 7 digits). IDs are keyed after non-ASCII replacement, so LÖ21-04 and LO21-04, or IDs differing only in a Unicode hyphen, became one key and were merged into one record or failed later as a colliding genus. Such pairs are now reported at input validation with both original strings, across DB and query tables. An ID repeated verbatim is still merged with a warning. IDs without any letter or digit are rejected as empty. The three test datasets give byte-identical outputs to 1.0.1, and the accession list sent to GenMine for terrei and sanghuangporus is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZTJs2CAFcHw9Ye4k7p9Mx
Besides setting the bitscore gap between ingroup and outgroup, the value discards every search hit at or below it before clustering (search.py), which the help text did not say. docs/parameters.md is regenerated from the parser. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZTJs2CAFcHw9Ye4k7p9Mx
12c6d66 handed V.list_FI and the FI dict to the outgroup and tree interpretation pools through module-level stores filled in the parent before the Pool was created. Only fork-started workers inherit those stores. Windows and macOS start workers by spawning, so every run below --verbose 3 (the default is 2) stopped at APPENDING OUTGROUP with KeyError: 'list_FI'. The bundled test presets use VERBOSE: 3, which runs the steps sequentially, so --test runs never reached this path. Fill the stores through the Pool initializer. On Linux the data is still inherited through fork; on spawn platforms it is sent once per worker instead of once per task. Checked on Windows by running the three test datasets with --verbose 2: they now finish, and every output is byte-identical to 1.0.1's sequential run apart from report.txt recording the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZTJs2CAFcHw9Ye4k7p9Mx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZTJs2CAFcHw9Ye4k7p9Mx
Make development identical to main
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.