perf: avoid redundant prefilters for complete ANN segments

[Gabriel39]

Unfiltered ANN searches scoped to complete index segments can build redundant row-ID allowlists. Pin all Lance crates to the merged development-branch revision 68c12dfd7efe02f90ad2d7f3a239a7eb64884e57, containing lance-format/lance#9599, the PQ scoring fix in lance-format/lance#9537, and ANN stage timings in lance-format/lance#9602. Align the direct OpenDAL dependency with Lance.

The complete-segment fast path preserves predicate, partial-coverage, deletion, stable-row-ID, and unindexed-tail behavior. The C API regression checks exact results and prefilter materialization counters across 16 combinations. The PQ regression compares unfiltered and all-row-filtered results for L2, Cosine, and Dot.

Verify that stage timings cross the existing dynamic statistics callback with the nanosecond unit. A reported zero duration is valid for short or uncontended stages, so the callback test checks metric presence and type without requiring every value to be positive. The C ABI is unchanged. Timers accumulate concurrent work and may overlap; they are not additive query latency.

The materializing cases also require both prefilter build/load timers to be present with the nanosecond unit. A temporary mutation removing the build timer failed as expected; the final test passed across all 16 combinations. All 443 regular tests, Clippy, and formatting were rerun for this test-only follow-up.

Validation on Linux x86_64 with Rust 1.98.1:

• cargo test --locked: all 443 tests passed, including 361 C API tests.
• cargo test --locked --test compile_and_run_test -- --ignored: all three opt-in tests passed. These compile and execute real C and C++ API consumers against the shared library, plus a C consumer against the static archive. The static test verifies that both ordinary and shared-runtime OSS opens reach a local HTTP server from fresh processes, without relying on whole-archive linking.
• cargo clippy --locked --all-targets -- -D warnings and cargo fmt --all --check passed.

The native result above is local Linux validation; macOS remains covered by the existing PR CI job. No production-data latency measurement is claimed.

[yanghua]

Please run the C/C++ static-link tests for this dependency update.

This changes the exact OpenDAL version and moves the complete Lance dependency graph from 11.x to 13.x, both of which are linked into the exported staticlib. However, the PR description explicitly says the three opt-in C/C++ compilation tests were not run. Rust tests cannot detect missing native symbols, static initialization, or HTTP transport registration failures in an actual C/C++ consumer.

Please run cargo test --test compile_and_run_test -- --ignored on the supported target(s), or add the equivalent CI result, before merging.

[yanghua]

Avoid requiring every stage timer to be non-zero.

The test is intended to verify that dynamic metrics cross the C callback with the nanosecond unit, but value > 0 additionally assumes every stage consumes at least one observable clock tick. In particular, index_cpu_queue_wait_time can legitimately be zero when there is no queue contention, and short cached stages can also round to zero on some platforms.

Please assert that each metric is present with TimeNanoseconds; only require a positive value for a deliberately controlled operation, or aggregate repeated executions before checking positivity.

[Gabriel39]

Fixed in abba808. The callback regression now checks that each named metric is present with TimeNanoseconds and accepts zero durations. Exact-result and prefilter materialization assertions remain intact. All 443 regular tests, all three opt-in native tests, Clippy, and formatting passed on the merged Lance revision.

[Gabriel39]

Addressed review #89 (review) on abba808. The dependency now pins the merged Lance #9602 revision, 68c12dfd7efe02f90ad2d7f3a239a7eb64884e57.

On Linux x86_64 with Rust 1.98.1, cargo test --locked --test compile_and_run_test -- --ignored passed all three tests: C and C++ shared-library consumers compiled and executed successfully, and the static C consumer verified both ordinary and shared-runtime OSS opens from fresh processes. Both modes reached the local HTTP server; the archive was linked explicitly without whole-archive.

cargo test --locked also passed all 443 regular tests, and Clippy and formatting passed. The PR description now records this native validation instead of saying the opt-in tests were not run. macOS validation remains in the existing CI job.

[LuciferYang]

Three non-blocking observations from reviewing the dependency bump and the two new tests. All minor: two are about test-assertion strength, one about the pin target. The dependency pinning itself checks out (all crates on one rev, no stale rev left in the lock, and the segment-compat validation is still present upstream). Details inline.

[LuciferYang]

prefilter_build_time and prefilter_load_time are only ever asserted == 0 here, and the sum() runs over filter(name == ...), so a typo or an upstream rename makes the filter match nothing, the sum stays 0, and the assert passes vacuously. These two timers effectively aren't tested. The sibling counters prefilter_input_rows/prefilter_row_ids get a positive == 32 assertion in the full-snapshot test that pins their names; these two timers have none.

In a branch that does materialize (filtered, or !segmented), assert that the two metrics are present (.any(|(name, _, _)| name == ...)). That pins the names without relying on a specific timer value.

[Gabriel39]

Fixed in 9bd730a. Materializing cases now require both prefilter_build_time and prefilter_load_time to be present with TimeNanoseconds, without requiring positive durations. A temporary mutation dropping prefilter_build_time from the callback failed with the expected missing-metric assertion; after removing the mutation, all 443 regular tests, Clippy, and formatting passed.

[LuciferYang]

Requiring index_cpu_queue_wait_time > 0 across all 16 combinations looks brittle. With small test data on an idle machine the CPU-dispatch permit is usually available immediately and nothing queues, so this wait can legitimately be 0; on a faster CI box the assert could fail intermittently. index_partition_load_time has the same risk once the cache is warm.

The timers that wrap real work (search, distance_topk, result_materialize) are fine to keep at > 0. For the pure wait/load timers, asserting the metric is present rather than > 0 removes a cross-machine flake source. This depends on Lance's timing semantics, so it's worth confirming whether the timer can be 0 before relying on it.

[Gabriel39]

This was already addressed in abba808: the ANN timing loop checks presence and TimeNanoseconds only, including queue/load timers. It no longer requires positive values.

[LuciferYang]

350351e is the pre-squash branch head of the now-merged PR lance-format/lance#9602 (Gabriel39:dev/ann-search-profile); it isn't on lance main (diverged: ahead 3 / behind 4). All three fixes it carries (#9537/#9599/#9602) are already on main, so pinning to a main ancestor would give the squashed, reviewed state and keep the dependency on main. Durability isn't the concern here: refs/pull/9602/head keeps 350351e fetchable even if the source branch is deleted. This is just tidiness.

Minor: the description says "including the merged lance-format/lance#9602", but 350351e is the pre-squash head, and the squash commit 68c12df isn't an ancestor of it. Pinning to a main commit would make that line accurate too.

[Gabriel39]

This was already addressed in abba808. Cargo.toml and Cargo.lock now consistently pin 68c12dfd7efe02f90ad2d7f3a239a7eb64884e57, the merged #9602 commit on Lance main; the PR description was updated accordingly.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The prefilter timing review point is addressed: materializing cases now require both prefilter build and load metrics with the nanosecond kind, so an absent metric cannot pass a zero-sum assertion. The complete, unfiltered path still checks for no row-ID allowlist, and the 16-case regression retains exact-result and loader-count checks. The focused test passes on this head. Zero-duration stages remain valid; these overlapping timings are diagnostics rather than additive latency.

The pin remains the merged Lance ANN timing revision, carrying the segment-prefilter fix and screened 4-bit PQ scorer without a C ABI change. The previous revision's 361 C API tests and native C/C++ and static-consumer checks remain applicable because this commit changes only the regression. The merged pin's additional FTS and opt-in row-lineage paths have dedicated upstream regressions.