Close the three open gaps: release gate, dead dependencies, and ask's per-question traversal

[sroussey]

Closes #376, closes #377, closes #378. +330 / −74 across 13 files.

Three independent fixes in one commit. #376 and #377 are small and self-contained; #378 is the one worth the review time — it adds a column, a write path and a migration, and it carries two decisions the issue left open.

---

#376 — the release gate could publish from a red tree

Two defects in one line of package.json.

release-checks ran four of CI's five gates and skipped test. At d544bc64, with CI red for five days on packageManifest.test.ts, bun run release succeeded. The suite was the only gate that would have caught it.

It also ran format, which writes, rather than format-check. A tree failing the CI gate passed this one by being modified, and bunset --auto --push --commit --tag then committed the reformatting unreviewed — the opposite of what a gate is for.

-"release-checks": "bun run format && bun run lint && bun run typecheck && bun run build && bun run prepack-check"
+"release-checks": "bun run format-check && bun run lint && bun run typecheck && bun run build && bun run test && bun run prepack-check"

The guard the issue proposed to tighten alongside it lived in packageManifest.test.ts, which #379 deleted — so there is no guard here, by design. It is worth saying out loud rather than leaving implicit: nothing now asserts this list, and the reason is that the thing asserting it was rubber-stamping the gap.

#377 — seven runtime dependencies nothing imports

Residue of the re-founding: the code that imported a PDF reader, an NLP tokenizer and an XML parser went to embarc-data; the declarations stayed. 17 runtime dependencies → 10.

Dropped: cheerio-json-mapper, compromise, pdf2json, words-to-numbers, xml2js (20 MB, nothing in the graph pulls them in), @types/xml2js, plus @modelcontextprotocol/sdk and @huggingface/transformers-structured-output, which are already ordinary dependencies of @workglow/mcp and @workglow/cli and so resolve without sec declaring them.

This is a published package and build is --packages=external, so every install paid for all 20 MB.

#378 — ask traversed filing_document once per question

ask pre-indexes before every question, so after the first one the selection matches nothing — and a LIMIT that never fills has to visit every candidate to learn that. New filing_document.kb_indexed_at, with a partial index over the nulls, so a fully indexed corpus leaves that index empty.

Two decisions the issue left open, both taken toward safety. These are the ones to push back on if you disagree:

1. The anti-join against kb_document stays; the stamp only ever narrows it. A stamp that is missing or stale then costs a traversal rather than a document indexed twice — and indexing twice spends embedding calls, so the failure has to land on the cheap side. There is a case asserting an unstamped document already in the knowledge base is still skipped.
2. The backfill runs in the same db setup pass that creates the index, so the column never exists un-backfilled. This one is load-bearing: left empty, the partial index covers the whole table and the planner walks it in filing_date order issuing a random probe per row — the access path Once the corpus is fully indexed — the steady state for sec ask — the pre-index still traverses all of filing_document per question (342 ms at 300k rows), and the new (filing_date, accession_number) index makes that regime slower, not faster #378 measured at 11× slower than the scan it replaces. Adding the column without backfilling would have made the steady state worse, not better. The query reads the stamp only when the column is present, which is what says setup has run.

A re-conversion at a newer FILING_CONVERTER_VERSION resets the stamp to null, since the text the knowledge base holds has been replaced.

The partial index is raw DDL in setupAllDatabases beside the existing raw DDL there, because defineStorage's indexes takes plain column lists and cannot express a WHERE clause. Worth knowing as a small gap in the registry rather than a choice about this change.

---

Verification

bun run release-checks — the new gate, end to end — exit 0:

Test Files  151 passed | 3 skipped (154)
     Tests  1490 passed | 21 skipped (1511)
prepack-check OK (npm pack --json): 6 file(s), unpacked 938.12 KB.

Five new cases, each checked to fail for the right reason:

• EXPLAIN QUERY PLAN shows the steady-state selection reading filing_document_kb_unindexed, with no table scan and no temp B-tree
• the backfill stamps pre-existing indexed rows, and is idempotent across two runs
• an unstamped document already in the knowledge base is still skipped
• the stamp lands on a real run and not under --dry-run
• removing the write turns that last case red (reinstated and restored)

🤖 Generated with Claude Code

https://claude.ai/code/session_01BLxewpuWR1ZeGdt3XHvVDh

---

Generated by Claude Code