perf: adopt Lance full-snapshot prefilter optimization

[Gabriel39]

Summary

Explicitly selecting every fragment in a snapshot should not require scanning every indexed row ID before ANN/FTS search. Adopt the Lance v11 prefilter optimization and loader statistics from lance-format/lance#9460, upstreaming the dependency update used in apache/doris#68408.

Pin all Lance dependencies to db211492fc5cd9da5642d7234d9682de9be77c19. This revision includes lance-c's existing 356acb0 dependency, preserving vector segment compatibility validation. The lockfile changes only the 21 Lance source revisions; package versions, Arrow 58 dependencies, and the C ABI are unchanged.

Add a C API regression test with two fragments and both stable and physical row IDs. It checks exact IVF_FLAT results, verifies that full explicit and implicit scopes scan the same number of rows without loading an allow set, and checks that strict fragment subsets still load and enforce their row-ID prefilter.

Dependency and compatibility

lance-format/lance#9460 is still open; this pins its tested integration revision. The optimization can select the existing bulk scorer: 4-bit IVF_PQ candidate recall at a fixed refinement budget may change. The separate scoring fix in lance-format/lance#9537 on main is not included in this v11 update. See the upstream PR for the detailed compatibility discussion.

Validation

• The new C API test fails against the previous 356acb0 pin: an explicit full scope scans 69 rows versus 5 for the implicit scope. The same test passes with the new pin, including both row-ID modes and strict-subset counters.
• cargo fmt --all -- --check: passed.
• cargo clippy --locked --all-targets -- -D warnings: passed with Rust 1.91.0.
• cargo test --locked: 441 passed.
• cargo test --locked --test compile_and_run_test -- --ignored: all three passed (C, C++, and static-library OSS transport).
• Upstream integration: 84 prefilter tests and 18 segment commit/compatibility tests passed. Its workspace Clippy run is blocked by an existing diagnostic in an unchanged file; details are recorded in fix: skip redundant full-snapshot row-id prefilters and expose loader metrics lance#9460.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The pinned v11 change skips the redundant row-ID load only when an explicit fragment scope covers the manifest and there is no prefilter predicate. The C regression passes for both row-ID modes and retains strict-subset filtering; the existing deletion, overlay, and unindexed-fallback paths remain in place.

Complete scopes can now use the existing 4-bit IVF_PQ bulk scorer, which can change candidates and reduce recall at a fixed refinement budget. The separate scoring fix is merged on main but absent from this v11 pin. Recall-sensitive callers should benchmark this query shape and adjust refinement if needed.

[yanghua]

+1