fix(scanner): preserve requested projection in fast search

[lance-gatefixer]

fast_search() changed the requested projection, causing FTS and vector queries selecting only id to return an extra _rowid column.

Remove the row-ID projection mutation from fast_search(). Search nodes already produce the row IDs needed internally, and the take step preserves them. Explicit _rowid projections and with_row_id() continue to return row IDs. This also preserves deleted-row validation and makes scalar-query plans independent of whether project() runs before or after fast_search().

Regression coverage includes FTS and vector search, default and explicit projections, requested row IDs, and both row ID modes over two fragments. It checks result equality with normal search and recall 1.0, rejects deleted-row scans without requested row IDs, and compares scalar-query plans and results across both call orders.

Validation:

• cargo test -p lance fast_search --lib — 38 passed.
• cargo fmt --all — passed.
• cargo clippy --all --tests --benches -- -D warnings — passed.

Fixes #9696

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[zhangyue19921010]

The extra _rowid column is gone. However, I'd suggest removing the remaining line as well, rather than keeping it as an internal-only flag:

      self.projection_plan.physical_projection.with_row_id = true;                                                                                                                                                                                 

This line isn't needed. The vector and FTS search nodes already output _rowid. When the take step adds the missing columns, take_current also carries over the _rowid that is already in its input (projection.with_row_id |= has_row_id).

It defeats the include_deleted_rows check. During planning, include_deleted_rows is rejected when physical_projection.with_row_id is false, because a NULL _rowid is the only way to identify a deleted row.

It does extra work, and the plan depends on call order. For a scalar-index filtered query, calling fast_search() after project(["id"]) makes the plan read row IDs and then add an extra projection to drop them.

[lance-gatefixer]

Addressed in 66a4a99. Removed the physical row-ID flag; fast_search() now leaves projection unchanged while search nodes and take_current preserve internal IDs. Regression tests cover include_deleted_rows() validation and identical scalar plans/results across both call orders. All 38 fast-search tests, formatting, and Clippy pass.

[lance-gatekeeper]

✅ Gate recommendation: approve.

Removing the projection mutation fixes #9696 while search nodes and materialization retain the row IDs they need. The revision also addresses the deleted-row validation and call-order concerns: explicit row-ID requests still work, deleted-row scans require them, and scalar-query plans are independent of project()/fast_search() order.

[lance-gatefixer]

Blocked: the windows check fails on a PyTorch TorchInductor cache permission error outside the fast-search repair.

Head: 66a4a99e48b105f61388a52dbf3fdeb628af1c21. The failed Windows job fails in test_torch_index_with_nans[Legacy] with PermissionError: [Errno 13] when PyTorch opens a generated code-cache module; the suite reports 1 failed, 1683 passed, and 780 skipped.

I inspected the job log and failing call path. This test builds a Torch IVF-PQ index without calling fast_search(), while this PR changes only scanner projection behavior and regression coverage. The fetched main still matches the repair's base, so there is no base update to integrate. Local Linux testing cannot reproduce the Windows runner's cache permissions.

Please rerun the failed Windows job on this head with a fresh runner. If the error repeats, investigate the runner's TorchInductor cache permissions separately and rerun after correcting them.