[xingbowang]

Summary:

Why

Applications need to filter the logical merge result. Previously, compaction filters ran before merging and could remove individual operands before the merge operator evaluated them.

Behavior

The default remains filter-before-merge. Filters and factories can opt into merge-before-filter through GetMergeOrder(). The existing FilterV4 receives full values/entities, partial merge operands, or individual untouched operands when partial merge declines. Successful aggregates are offered once; changed operands are not merged again in the same pass.

Snapshot eligibility covers every retained operand. Stored values retain normal filtering rules, including sequence zero; aggregates containing ineligible operands are not filtered. New-order value/entity kPurge uses ordinary Delete semantics for input records, merge results, and later compactions. Unsupported operand decisions return errors.

Example

Counter merge operator (add) with a filter that removes counters below 10. The compaction input for one key is Merge(+3), Merge(+5), Put(4). Before: the filter sees the raw operands +3 and +5, removes both, and the output is Put(4); it never sees the merged value. After: the merge runs first (4 + 5 + 3 = 12) and FilterV4 sees kValue "12" and keeps it. If the base is not in the compaction, partial merge gives +8 and FilterV4 sees it as kMergeOperand.

Implementation

MergeHelper stages outcomes and eligibility; CompactionIterator handles filtering, removal and range skips, including the advanced cursor after a full merge. Production factory/order checks run outside the DB mutex. Open and CF creation reject new-order factories selecting non-compaction creation; flush/recovery/mempurge retain runtime guards. Repair explicitly checks the table builder's IOStatus while preserving earlier error precedence. Purge normalization has a ticker.

Compatible services advertise support and the worker acknowledges its actual filter order. Missing capability, service kUseLocal, and results without a matching acknowledgment fall back locally before output import; old workers ignore the new request field and return no acknowledgment, so they fall back too. Real errors remain errors. Service integration is supplied separately and activated by the normal RocksDB release.

Safety and limitations

FilterV4 and value types remain unchanged; there is no persisted SST-format change, new CF option, or C/Java binding. New virtual methods and job metadata require rebuilding applications, filters, factories, services and workers against matching headers/libraries. Binary rollback requires a compatible rebuild. Normalized purge cannot be mixed with SingleDelete for the same keys.

Integrated BlobDB is supported. SnapshotChecker, timestamps and stacked BlobDB are unsupported; SnapshotChecker and timestamps remain follow-ups. Merge grouping can change callback inputs and policy outcomes; existing callback lifetime/exception rules apply.

Focused native tests cover merge/filter and remote routing behavior. Full-suite, current sanitizer and benchmark validation are unavailable. Five established immutable-baseline checked-status failures remain unresolved and unwaived. No performance or production rollout claim is made.

Differential Revision: D128313536

[meta-codesync]

@xingbowang has exported this pull request. If you are a Meta employee, you can view the originating Diff in D128313536.

[github-actions]

⚠️ clang-tidy: 11 warning(s) on changed lines

Completed in 964.4s.

Summary by check

Check	Count
clang-analyzer-optin.core.EnumCastOutOfRange	4
clang-analyzer-optin.performance.Padding	1
cppcoreguidelines-special-member-functions	1
modernize-use-emplace	4
performance-inefficient-vector-operation	1
Total	11

Details

db/compaction/compaction_iterator_test.cc (4 warning(s))

db/compaction/compaction_iterator_test.cc:640:16: warning: use emplace_back instead of push_back [modernize-use-emplace]
db/compaction/compaction_iterator_test.cc:643:14: warning: use emplace_back instead of push_back [modernize-use-emplace]
db/compaction/compaction_iterator_test.cc:687:18: warning: use emplace_back instead of push_back [modernize-use-emplace]
db/compaction/compaction_iterator_test.cc:704:20: warning: use emplace_back instead of push_back [modernize-use-emplace]

db/db_compaction_filter_test.cc (3 warning(s))

db/db_compaction_filter_test.cc:36:7: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/db_compaction_filter_test.cc:1070:38: warning: The value '99' provided to the cast expression is not in the valid range of values for 'MergeOrder' [clang-analyzer-optin.core.EnumCastOutOfRange]
db/db_compaction_filter_test.cc:1075:53: warning: The value '99' provided to the cast expression is not in the valid range of values for 'MergeOrder' [clang-analyzer-optin.core.EnumCastOutOfRange]

db/flush_job_test.cc (1 warning(s))

db/flush_job_test.cc:190:9: warning: class 'Filter' defines a non-default destructor but does not define a copy constructor, a copy assignment operator, a move constructor or a move assignment operator [cppcoreguidelines-special-member-functions]

utilities/blob_db/blob_db_test.cc (3 warning(s))

utilities/blob_db/blob_db_test.cc:1003:25: warning: The value '255' provided to the cast expression is not in the valid range of values for 'MergeOrder' [clang-analyzer-optin.core.EnumCastOutOfRange]
utilities/blob_db/blob_db_test.cc:1004:10: warning: Excessive padding in 'struct Case' (8 padding bytes, where 0 is optimal). Optimal fields order: direct_order, factory_order, direct, factory, invalid_argument, not_supported, consider reordering the fields or adding explicit padding members [clang-analyzer-optin.performance.Padding]
utilities/blob_db/blob_db_test.cc:1093:11: warning: The value '255' provided to the cast expression is not in the valid range of values for 'MergeOrder' [clang-analyzer-optin.core.EnumCastOutOfRange]