Skip to content

PARQUET: Account for custom PageIndexProvider memory - #11057

Draft
zhangxinyao88 wants to merge 1 commit into
apache:mainfrom
zhangxinyao88:codex/arrow-rs-11028-page-index-memory
Draft

PARQUET: Account for custom PageIndexProvider memory#11057
zhangxinyao88 wants to merge 1 commit into
apache:mainfrom
zhangxinyao88:codex/arrow-rs-11028-page-index-memory

Conversation

@zhangxinyao88

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

ParquetMetaData::memory_size() currently accounts for page-index memory only when the provider is the built-in PageIndex. Custom PageIndexProvider implementations are therefore reported as zero even when they own cached index data.

What changes are included in this PR?

  • Add a default PageIndexProvider::heap_size() -> usize method returning zero.
  • Have the built-in PageIndex delegate to its existing private HeapSize implementation.
  • Account for the concrete provider size, Arc allocation overhead, and provider-reported heap allocation in ParquetMetaData::memory_size().
  • Add regression coverage for a custom provider that reports heap usage.

Are these changes tested?

  • cargo test -p parquet file::metadata --no-default-features
  • cargo check -p parquet --example custom_page_index --no-default-features --features arrow
  • rustfmt --check on modified files.

This change was developed with AI assistance. I reviewed the implementation and verified the built-in accounting path, the custom-provider regression test, and source compatibility for the existing custom-provider example.

Are there any user-facing changes?

Yes. Custom PageIndexProvider implementations can opt into ParquetMetaData::memory_size() accounting by overriding heap_size(). Existing implementations remain source-compatible and report zero additional heap usage by default.

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Sep 11, 2026
@etseidl

etseidl commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Thanks @zhangxinyao88, but I think a PR for this issue is premature. I think we first need to decide on the approach we want to take before we start implementing solutions.

@zhangxinyao88

Copy link
Copy Markdown
Contributor Author

Thanks, that makes sense. I’ll mark this as draft until we’ve agreed on an approach in #11028.

@zhangxinyao88
zhangxinyao88 marked this pull request as draft September 12, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support reporting heap size of custom PageIndexProvider

2 participants