fix: keep the kb_indexed_at stamp honest, and out of a dry run

[sroussey]

Three defects around filing_document.kb_indexed_at, the cache the index selection reads as a narrowing of its kb_document anti-join.

1. A knowledge base rebuilt from empty was never repopulated (critical)

The stamp was only ever set, never cleared. getSecKnowledgeBase tells the operator to drop kb_document, kb_chunk and kb_index and re-run sec index when the embedding model changes — and IndexFilingSectionsTask.execute opens the knowledge base before it selects, so the DDL re-creates kb_document empty. kbDocumentTableExists() is then true, the d.kb_indexed_at IS NULL clause switches on, and the stamps from the previous index exclude the whole corpus. The anti-join on its own would have selected every document. The run reported indexed: 0 and sec ask answered from an empty index with no error; db setup could not heal it, because syncKbIndexedStamp only ever set stamps.

The stamp now moves in both directions:

• syncKbIndexedStamp clears the stamp of any document with no surviving kb_document row, and clears the column outright in the branch where the table is gone.
• Opening the index clears every stamp when the index holds no documents (clearKbIndexedStampIfIndexEmpty), which is the check the rebuild path needs and is cheap enough to run on every open: one LIMIT 1 probe wherever the index holds anything, against the full reconciliation's probe per stamped row. The scan it falls through to only happens where nothing is indexed — a run that is about to embed a corpus.

Both directions err toward a traversal (a missing stamp costs one; the anti-join still prevents a double index), never toward a document that is never offered again.

2. --dry-run wrote to the database (high)

syncKbIndexedStamp(getDb()) sat outside any guard in setupAllDatabases, so sec db setup --dry-run and sec setup --dry-run issued a real UPDATE filing_document SET kb_indexed_at = … and a real CREATE INDEX. Raw SQL goes around the ReadOnlyTabularStorage wrapper. The isDryRun() bail now lives inside the functions rather than at the call site, so a future caller cannot bypass it.

3. A plan assertion that could never fail (high)

expect(detail).not.toContain("SCAN d\n") tested for a newline that EXPLAIN QUERY PLAN details never contain, so the test's "no table scan" claim was unenforced.

Worth noting for reviewers: the suggested not.toMatch(/\bSCAN d\b/) would have failed the passing case — SQLite reports a walk of the partial index as SCAN d USING INDEX filing_document_kb_unindexed. So the assertion is now the whole plan: one step, and it walks the partial index, plus a check that no step reaches the table itself and none sorts into a temp B-tree. A companion test drops the partial index and pins what is actually lost — not a table scan, since filing_document_filing_date_accession_number still serves the ORDER BY, but an index over every row instead of only the unindexed tail.

Tests

src/config/kbIndexedStamp.sqlite.test.ts gains four tests; each fails on main (verified by reverting the two source files and re-running):

• clears a stamp the knowledge base no longer backs
• clears every stamp when the knowledge base tables are gone
• re-offers every document after the index is dropped and rebuilt empty — the traced failure, end to end through getSecKnowledgeBase and selectDocumentsToIndex
• writes nothing under a dry run

Plus leaves the stamps alone when the index still holds documents, which pins the other half of the rule, and the reworked plan tests.

Verification

bun run format && bun run format-check   # All matched files use the correct format
bun run lint                             # exit 0
bun run typecheck                        # exit 0
bunx vitest run src/config src/task/kb src/kb
  Test Files  22 passed | 1 skipped (23)
       Tests  155 passed | 6 skipped (161)
bunx vitest run src/task/document/selectFilingsToConvert{,.sqlite}.test.ts
  Test Files  2 passed (2)   Tests  16 passed (16)

🤖 Generated with Claude Code

https://claude.ai/code/session_019fRkYZYfNTzqATS6DA7y7y

---

Generated by Claude Code