Skip to content

fix(mem_wal): stream conditional file promotion - #9704

Open
jackye1995 wants to merge 6 commits into
lance-format:mainfrom
jackye1995:jack/memwal-streamed-promotion
Open

jackye1995 wants to merge 6 commits into
lance-format:mainfrom
jackye1995:jack/memwal-streamed-promotion

Conversation

@jackye1995

@jackye1995 jackye1995 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • stream fixed-name MemWAL promotions through the client on native S3 instead of using server-side copy
  • preserve writer fencing with conditional single-part and multipart destination creation
  • retain the existing native conditional-copy path for GCS, Azure, local, OpenDAL, and custom stores
  • treat an equal-size existing destination as an idempotent retry without overwriting it
  • pin Add PutMultipartOptions::mode for conditional multipart completion apache/arrow-rs-object-store#863 until its conditional multipart API is released

Validation

A full 1,161,228-row LAION-1M Blob v2 MemWAL campaign completed on S3 with one discarded prewarm and one measured trial. The measured run completed at 626.9 rows/s, passed all 101 key and image checksum samples, and completed the formerly failing 3.49-GiB conditional SSTable promotion without CopyObject or UploadPartCopy.

Route fixed-file promotion through client-streamed create-if-absent uploads so large S3 flushes avoid server-side copy timeouts without weakening writer fencing.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-python Python bindings A-java Java bindings + JNI A-deps Dependency updates A-encoding Encoding, IO, file reader/writer bug Something isn't working labels Oct 4, 2026
Clean up conditional multipart uploads when destination creation is rejected.
Apply the same equal-size idempotency rule to conditional local copies as cloud multipart promotion.
lance-gatekeeper[bot]

This comment was marked as outdated.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Oct 4, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Oct 4, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The backend fallback and wrapper fix address both reported regressions. Native S3 can use conditional streaming while unverified stores retain their established conditional-copy path. No remaining blockers found.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 4, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates A-encoding Encoding, IO, file reader/writer A-java Java bindings + JNI A-python Python bindings bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant