DX-124448: [Parquet] Add first-page dictionary benefit check

[bodduv]

What

Adds WriterProperties::Builder::enable_dictionary_benefit_check() (default off). When enabled, the first data page of each dictionary-encoded column chunk decides whether dictionary encoding is kept, using parquet-java's rule (FallbackValuesWriter / DictionaryValuesWriter.isCompressionSatisfying):

keep dictionary  iff  encoded indices of the first page + dictionary size  <  first page size in PLAIN encoding

If the first page fails the check, the writer falls back through the existing FallbackToPlainEncoding(): the dictionary page and the first page stay dictionary-encoded, and the remaining pages are PLAIN. Without the check (today's behaviour), the dictionary is only abandoned when it reaches dictionary_pagesize_limit.

Why

Dremio's Arrow C++ Parquet writer (DX-124448, dremio/dremio#29048) was 0.86× of parquet-java on a CrowdStrike-style CTAS with 60 high-cardinality 16-character string columns. Footer and page dumps showed Arrow C++ dictionary-encoding each column chunk until the 2 MiB limit (about 105K values, roughly half a 211K-row chunk) before falling back. That cost +10.6% bytes per row and the hashing and index-encoding CPU. parquet-java abandons the dictionary after the first page.

How

• properties.h: the new builder option and its getter, copied in Builder(const WriterProperties&).
• column_writer.cc:
  • ColumnWriterImpl::AddDataPage records the exact encoded values size of each page.
  • TypedColumnWriterImpl accumulates the first page's PLAIN size while the check is pending. That is non-null count × width; for BYTE_ARRAY it is count × 4 plus the encoder's existing unencoded-byte counter, and for FLBA count × type_length. The accounting matches dict_encoded_size().
  • The check runs right after the chunk's first page is cut in CommitWriteAndCheckPageLimit.
• Every row group still starts a fresh dictionary, and dictionary_pagesize_limit still applies after the check passes.

Known limits

• A column chunk that ends before its first page is cut is never checked: its only page is written at Close().
• Values written as an Arrow DictionaryArray (WriteArrowDictionary) are not checked, because their PLAIN size is unknown.
• Unlike parquet-java, the first page is not re-encoded as PLAIN.

Tests

• column_writer_test.cc:
  • DictionaryBenefitCheckFallsBackAfterFirstPage: unique values give exactly 1 dictionary-encoded data page, and the rest are PLAIN.
  • DictionaryBenefitCheckKeepsBeneficialDictionary: 10 distinct values give all pages dictionary-encoded.
  • Both run for every dictionary-eligible type, for Parquet 1.0 with V1 data pages and for 2.6 with V1/V2 data pages.
• properties_test.cc: the default is off; the setters work and survive the builder copy.
• Run on Linux arm64 (manylinux2014, GCC 10): parquet-internals-test (TestWriterProperties.*), the full parquet-writer-test (330 tests), and parquet-arrow-reader-writer-test *Dictionar*. All passed.
• clang-format 14.0.6 makes no changes.

🤖 Generated with Claude Code