feat(compaction): add Dataset::column_layout_stats

[LuciferYang]

What

Dataset::column_layout_stats() returns column-layout stats for each fragment. It reads only fragment metadata, so it's cheap to call and does no I/O. For each fragment it reports:

• live_file_count: how many data files hold at least one column of the dataset schema;
• file_sizes: each data file's recorded size, None when the manifest has none;
• fields_per_file: how many schema fields each data file holds;
• tombstoned_field_ratio: the share of the fragment's field slots that hold no live data, either tombstoned by a column update or left by a dropped column. A fragment rewrite reclaims them;
• overlay_count: how many overlay files the fragment carries.

A file counts as live when it holds a field id of the current schema. A file holding only tombstones, only dropped columns, or only a spilled row lineage sequence does not, since no read touches it. The reserved lineage ids aren't user columns, so they count on neither side of the ratio.

Why

This is the planning input for deciding which fragments to repack. Repeated add_columns backfills drive up a fragment's data-file count, and that decision should be observable rather than buried in a heuristic. The sizes and per-file field counts let a custom planner tell small backfill files from large ones, and the dead slot ratio shows how many of a fragment's field slots no read uses.

Testing

• column_layout_stats_counts_files_per_fragment writes a two-fragment dataset, asserts one data file each, runs add_columns, and asserts two live files per fragment with one field each, known sizes, and no dead slots.
• column_layout_stats_counts_dropped_columns_as_dead_slots drops one of two columns and asserts one live field and a ratio of 0.5.
• column_layout_stats_skips_files_without_schema_columns adds a tombstone-only file, a dropped-column file, and a tombstone-plus-row-id file to a fragment, and asserts only the real data file is live and the ratio is 0.75 (the row id slot isn't counted).

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[lance-gatekeeper]

✅ Gate recommendation: approve.

This gives horizontal compaction planning a cheap, observable per-fragment file count. The count follows the current manifest cleanup rules for tombstoned and dropped fields, and the focused two-fragment backfill test passes. I found no material issue with this standalone API.

[LuciferYang]

Context, kept out of the description above: this is the "compaction should expose stats instead of deciding behind a black box" surface, from the #9604 stack. It stands alone and depends on nothing else in that stack.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The count now follows the current schema, so files retained only for dropped columns or row lineage do not inflate the repacking signal. Both layout-stat tests pass, including the new regression case. This remains a useful planning API that reads manifest metadata only.

[lance-gatekeeper]

✅ Gate recommendation: approve.

File sizes and the dead slot ratio add useful planning signals using metadata alone. The three focused tests pass, covering backfills across two fragments, dropped columns, unknown sizes, and exclusion of lineage slots. I found no material issue.