Improvements surfaced during the GAIN challenge build + blind runs. None are in the frozen
v1.0.1 submission tag; they land on master for a future v1.0.2.
- Version in the
--helpheader.--versionworks but the header doesn't show it. One line:@app.callback(help=f"muck {__version__} — …")insrc/muck/cli.py(import__version__at top). - Document
text_fieldsflattening prominently. Agents misread this 3× — concluding nested fields (foreign_entities,conviction_disclosures,covered_position) "aren't in the index" and either skipping findings or pre-flattening the corpus. The truth: with notext_fields(or a plain nested key intext_fields), muck flattens the whole record into searchable + citable text; the[]array-path syntax works only instructured_fields. Add a bold callout inskill/SKILL.md(configure step) and near the top ofskill/references/ADAPTERS.md. muck peekshould suggest array-path fields.peek.pysuggested_field_maponly emits scalar leaves; for arrays-of-objects it should proposefield[].sub as aliasso an agent designing a field map for a nested corpus gets the aggregation fields handed to it.- A
--dir <corpus>flag so muck isn't cwd-dependent (complements the v1.0.1 child-dir hint in_resolve). Lets an agent anchor without relying on the shell's cwd surviving interrupts.
- Embeddings are the long pole. Batch larger + multiprocess across cores in
potion_embedder.embed/pipeline.run_indexflush_vectors. - Drop
raw_jsonbloat. Storing the full original record per doc is a large share of the 20 GB; make it optional (--no-raw/ config) or store only when needed formuck read --raw. - Cap / incrementally build entity edges. Co-occurrence edges scale fast (36k docs → 765k edges;
715k docs → 4.5M edges). Add a per-node degree cap or a
--no-edgesoption for huge corpora (extract/entities.pybuild_entities). - A keyword-only fast build. A
muck build --no-embedflag (currently needs a config edit) so a reproducer can index a large corpus in minutes; the deterministic playbooks (anomalies/aggregate/ boilerplate/grep) don't need embeddings. - Wire turbovec (already an opt-in store seam) for ANN search at multi-million-chunk scale.
- Make structured figures span-citable. Option to also render a chosen structured field (e.g.
income) intodocuments.textso the number passes the citation guard, not just its record. - Cross-source entity join + Tier-3 relationship graph. The deeper ceiling: wire the currently-
orphaned Splink
reconcileclusters into the query path, and giveentity_edges.edge_typereal typed predicates (only'co_doc'today) — enabling say-vs-pay, money-flow A→B→C, donor-is-client triangles that no single record contains.