Meta: facet-count correctness & coherence (counts beyond the single-filter cube)

[rdhyee]

Why this issue exists

Facet-count correctness keeps surfacing as separate bug reports (#296, #304, and others from collaborators), but they share one root cause and one fix lever. This is the higher-level tracker so the team can see the whole picture in one place and not re-diagnose each report from scratch.

It effectively succeeds #234 (now closed) for the counts dimension: #234's "coherent filter semantics across all surfaces" roadmap shipped, but the count-coherence frontier outlived it.

The through-line (plain English)

The Explorer can show correct, cross-filtered facet counts instantly for exactly one active filter — that case is served by a precomputed lookup ("the cube", #290/#298). Everything beyond one filter currently falls off that fast path:

• More than one filter at the zoomed-out view → other facets revert to their unfiltered global totals (this is Interactive Explorer: Facet counts wrong when filtering by multiple facet values #304).
• Viewport-scoped counts, active search, and flat mode also fall through to live paths with their own rough edges.

So the mental model is: the single-filter cube is the spine; the frontier is "counts when more than one thing is true at once."

Crucially, the map/point filtering is correct in all these cases — it's specifically the count numbers that mislead. And zooming in makes counts correct (the live membership path cross-filters by all active dims). The breakage is specific to the global/zoomed-out view.

The shared fix lever

We already shipped the tool to fix this: the sample_facet_masks bitmask (#299) computes multi-filter results in one ~6M-row columnar scan, instead of the ~39M-row sample_facet_membership scan that originally forced the global multi-filter count path to be stubbed to the baseline. The plan is to un-stub the global multi-filter count path on top of the masks.

Children / related

• Interactive Explorer: Facet counts wrong when filtering by multiple facet values #304 — facet counts wrong when filtering by multiple facet values (the live bug; keep riffing there). ← primary
• explorer: facet counts — remaining work after #157/#226 #230 — facet counts: remaining work after globe: Phase 1 unification — Specimen Type filter, SKOS labels, portable predicate, v2 facets #157/Explorer: unify click semantics — click opens detail card, external link is one-hop-away #226 (the concrete leftovers list)
• explorer: material facet semantics — single "first non-root" value vs membership ("anywhere in array") counts #276 — material facet semantics: single "first non-root" value vs membership ("anywhere in array") counts (a semantics decision that shapes what the counts even mean)
• explorer: clearing a search does not deterministically restore viewport-scoped counts (needs repro) #286 — clearing a search doesn't deterministically restore viewport-scoped counts (needs repro)

What "done" looks like

• Facet counts reflect all active selections (multi-filter), at any zoom, served fast (masks-backed) rather than reverting to baseline.
• A single, stated semantics contract for what a count means (resolves explorer: material facet semantics — single "first non-root" value vs membership ("anywhere in array") counts #276), applied consistently across map / table / facets.
• The individual children above closed against that contract.

cc @ekansa @akthom — filing this so the count issues you've reported have one coordinating home; the specific bug stays in #304.

[rdhyee]

Plan of record (Claude + Codex, joint)

Full write-up committed to the repo: PLAN_305_facet_counts.md (code citations, native benchmarks, SQL, validator gates). Condensed version below.

Core idea — count per node FROM the #299 bitmasks, no 39M-row membership scan

For a count dimension D over a qualifying pid set Q: bitwise-join Q against facet_node_bits (≤63 rows/dim) and GROUP BY concept_uri:

... JOIN node_bits nb ON nb.facet_type='<D>'
    AND (s.<D>_mask & (1::BIGINT << nb.bit_index)) <> 0
GROUP BY nb.concept_uri

Masks are one row/pid → COUNT(*) = distinct pids (no COUNT(DISTINCT) needed); ancestor closure → parent ≥ child for free. Semantics: OR within a dim, AND across dims, exclude the target dim (matches the existing cube + UI contract).

Feasibility — confirmed by native DuckDB benchmarks (real 6M-pid data)

All three tree dims via one 56-node join: ~0.69s; Eric's Material+Specimen-Type case: ~0.02s. Worst case ≈ 336M cheap bit-tests → WASM cold+warm benchmark stays a hard release gate.

Two firm rules

• Source → add as a plain VARCHAR column to a new complete per-pid index (pid, source, 3 masks, build_id, schema_version), not a mask and not a semi-join.
• Never fall back to the unfiltered baseline under active filters — that is this bug. On miss/error show "—"/unavailable/stale, and apply all dimension counts atomically after one stale-guard check.

Prerequisite data bug → #306

The current masks are built from membership, so they omit ~29,917 located samples with no tree membership (6,026,242 vs 5,996,325). The new index must start from samp_geo (zero-mask those pids). Filed as #306; lands in Phase 1.

Phases

1. Artifact + semantics — complete per-pid index (incl. Data: sample_facet_masks omits ~29,917 located samples with no tree membership (built from membership, not samp_geo) #306 fix + source), fingerprint the pid/source universe, AI-free validator, publish under a new immutable tag, ratify the count contract as membership/"anywhere in tree" (explorer: material facet semantics — single "first non-root" value vs membership ("anywhere in array") counts #276 — already shipped semantics).
2. Fix Interactive Explorer: Facet counts wrong when filtering by multiple facet values #304 at global view — generalized direct-control filter snapshot (today's describeCrossFilters() zeroes tree selections at global view = root cause); route multi-filter counts through the histogram; keep the single-filter cube as a pure optimization; benchmark cold+warm WASM; unavailable-not-baseline on failure.
3. Extend (not unify yet) — add viewport (padded bbox) + search (search_pids), characterization-test vs the membership path, then retire it.
4. Verify + clean up — predicate-gen unit tests, validator in CI, Playwright (Eric's URL, multi-value, source+tree, impossible combos, global↔viewport, superseded requests), remove the baseline fallback, Codex review, staging verify, smoke gate.

Top risks

1. WASM latency / connection starvation (unmeasured in-browser).
2. Stale/incomplete index generations (Data: sample_facet_masks omits ~29,917 located samples with no tree membership (built from membership, not samp_geo) #306 + membership-only build_id).
3. Honesty & async state-transition failures (baseline-as-truth + non-atomic repaint).

cc @ekansa @akthom