feat(compaction): add a pluggable CompactionExecutor

[LuciferYang]

What

compact_files gets a pluggable executor next to the pluggable planner it already has.

• CompactionExecutor is an object-safe trait with one method, execute(&Dataset, TaskData, &CompactionOptions) -> Result<RewriteResult>. It runs one task of a CompactionPlan.
• DefaultCompactionExecutor runs a task the way compaction always has, through rewrite_files.
• compact_files_with_executor(dataset, remap_options, planner, executor) plans with the planner, runs every task through the executor, and commits the results with commit_compaction. A caller can use it to run tasks somewhere else (for example on a cluster) or to wrap the default executor.
• compact_files_with_planner and CompactionTask::execute now go through DefaultCompactionExecutor.

Behavior

No change. DefaultCompactionExecutor::execute calls the same rewrite_files. compact_files_with_executor keeps the old compact_files_with_planner shape: run the tagged-FRI check, return early on an empty plan, clone a read snapshot, buffer_unordered over the tasks at num_threads concurrency, try_collect, then commit_compaction.

Tests

compact_files_runs_tasks_through_the_given_executor passes a counting executor that delegates to the default one, and checks that every planned task went through it and that the commit kept every row.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The shared execution path preserves vertical compaction's snapshot, bounded task execution, and single rewrite commit, while giving the planned horizontal path separate task and result types. Focused local and distributed compaction tests passed on this revision.

[lance-gatekeeper]

✅ Gate recommendation: approve.

This revision changes only a documentation link. The executor and committer still preserve vertical compaction's snapshot, bounded task execution, and single rewrite commit; the focused local and distributed tests from the previous revision remain applicable.

[LuciferYang]

Context, kept out of the description above: this is the pluggable executor/committer refactor from the #9604 stack, the "make compaction pluggable first" step, split out to review on its own. Horizontal compaction plugs in as a second CompactionExecutor/CompactionCommitter pair in a follow-up. #9604 has the full picture with both executors present and the direction discussion.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The revised documentation matches the internal driver's scope. The executor and committer preserve the existing vertical compaction flow, and the previously passing local and distributed compaction tests remain applicable.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The public executor hook follows the requested pluggable planner/executor direction in #9291. It retains the existing snapshot, bounded task execution, and rewrite commit while allowing callers to supply an executor. The custom-executor, local compaction, and distributed compaction tests passed on this revision.

[LuciferYang]

Closing. #9604 dropped this trait after review (#9604 (comment)): CompactionTask::execute dispatches by task kind instead, and nothing needed a custom executor.