perf(dataset): serve read_version_transaction from the held or cached manifest

[summer-exa]

Summary

Dataset::read_version_transaction(version) (→ read_transaction_by_version → Python LanceDataset.read_transaction(version)) always did resolve_version_location + a full read_manifest (download + protobuf decode of the whole manifest) just to learn transaction_section/transaction_file and the timestamp — even when the handle was already checked out at version or the session cache held that manifest. On orion.lance (~4.8M fragments) a manifest is ~1.2 GB and takes 25–45 s from S3, so the silk_node committer's idempotency window (checkout_version(latest) then read_transaction(v) for each unseen v) paid a full manifest download per version.

New branch order in read_version_transaction:

 pub async fn read_version_transaction(&self, version: u64) -> Result<VersionTransaction> {
+    if version == self.manifest.version {
+        // in-memory manifest + same TransactionKey cache path as read_transaction()
+        return Ok(VersionTransaction {
+            version,
+            timestamp: self.manifest.timestamp(),
+            transaction: self.read_transaction().await?,
+        });
+    }
     let manifest_location = self.commit_handler.resolve_version_location(..).await?;   // LIST/HEAD, not the body
-    let manifest = read_manifest(&self.object_store, &manifest_location.path, manifest_location.size)
-        .await.map_err(NotFound -> DatasetNotFound)?;
+    let cached = if manifest_location.size.is_some() {          // same guard as Dataset::get_manifest
+        self.metadata_cache.get_with_key(&ManifestKey { version, e_tag: manifest_location.e_tag.as_deref() }).await
+    } else { None };
+    let manifest = match cached {
+        Some(m) => m,
+        None => Arc::new(read_manifest(..).await.map_err(NotFound -> DatasetNotFound)?),
+    };
     // branch-mismatch check, read_transaction_from_storage, VersionTransaction { .. } — unchanged
 }

• Step 1 (own version): no resolve_version_location, no read_manifest. Reads/populates the TransactionKey cache exactly like read_transaction(); the "no cache pollution" guarantee still holds for other versions.
• Step 2 (cached manifest) was feasible: Dataset::get_manifest/checkout_version populate the shared session cache under ManifestKey { version, e_tag }, and resolve_version_location returns the same version/e_tag (a HEAD/LIST, not the body), so a cache hit gives the manifest and the location that read_transaction_from_storage needs. The lookup is guarded by manifest_location.size.is_some(), mirroring get_manifest, so a location without size (no trustworthy e_tag) skips reuse rather than guessing. Only get_with_key is used — no manifest is inserted for historical versions.
• Step 3 fallback unchanged: branch-mismatch check, DatasetNotFound for missing versions, timestamp from the manifest.
• Doc comment updated (it previously promised "no session cache is read or written").
• read_transaction_by_version and python/src/dataset.rs are untouched — no signature change, no Python rebuild needed.

Tests (rust/lance/src/dataset/tests/dataset_transactions.rs)

Manifests are inflated with a 256 KiB config value (MANIFEST_PADDING_BYTES) so a full manifest read is distinguishable from a ranged transaction read. Because the local tracking store books a HEAD as a read_bytes of the full object size, cache-path assertions use ranged_read_bytes(&IoStats) (sum of IoRequestRecord.range lengths, i.e. bytes actually transferred).

test_read_version_transaction_same_version_skips_manifest_read (4 fragments, latest = v5):

scenario	assertion	observed
read_transaction() on a fresh handle (yardstick)	read_bytes < 64 KiB, < manifest_size	4096 B, 1 IOP (manifest 262,732 B)
same-version read_version_transaction(latest)	read_iops == yardstick.read_iops, read_bytes == yardstick.read_bytes, read_bytes < 64 KiB, < manifest_size; result == read_transaction() and == fresh-open result	4096 B, 1 IOP
second read_transaction() after the above	read_iops == 0 && read_bytes == 0 (TransactionKey cache populated)	0
historical handle (v1) → read_version_transaction(latest) (fallback)	ranged_read_bytes > 256 KiB; result == same-version result	266,704 B ranged (529,436 B incl. HEAD)
read_version_transaction(9999)	Error::DatasetNotFound	—
checkout_version(3) → read_version_transaction(3)	read_bytes < 64 KiB; == read_transaction()	4096 B
old handle (v3) → read_version_transaction(latest)	== latest transaction/timestamp	—
different handle, same Session, after the first handle checked out v3 → read_version_transaction(3)	ranged_read_bytes <= 4 KiB (cached manifest reused)	4096 B ranged (266,670 B incl. HEAD)
different handle, fresh Session → read_version_transaction(3)	ranged_read_bytes > 256 KiB (uncached, full read)	266,546 B ranged

test_read_version_transaction_external_transaction_file extended: same-version read of a manifest with transaction_file (no inline section) returns the transaction, read_iops >= 1 (external file), and no request touches the manifest path.

Verification

$ cargo fmt --all                                                # clean
$ cargo clippy --all --tests --benches -- -D warnings             # Finished `dev` profile, no warnings
$ cargo test -p lance dataset::tests::dataset_versioning
test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 2462 filtered out
$ cargo test -p lance dataset::tests::dataset_transactions
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 2466 filtered out
$ cargo test -p lance --doc dataset
test result: ok. 23 passed; 0 failed; 23 ignored; 0 measured; 9 filtered out

Rollout

not live until a fork release 9.0.1+exa.9 and a monorepo pin bump; neither is in this PR.

Link to Devin session: https://app.devin.ai/sessions/455f5722a6ff4f9bbe4cafaf06a4ab84
Open in Devin Desktop: https://app.devin.ai/desktop/session/455f5722a6ff4f9bbe4cafaf06a4ab84?variant=devin
Requested by: @summer-exa

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring