Skip to content

perf(parquet): fuse consecutive filters on the same projection - #10859

Draft
haohuaijin wants to merge 2 commits into
apache:mainfrom
haohuaijin:parquet-same-projection-filter-fusion
Draft

perf(parquet): fuse consecutive filters on the same projection#10859
haohuaijin wants to merge 2 commits into
apache:mainfrom
haohuaijin:parquet-same-projection-filter-fusion

Conversation

@haohuaijin

@haohuaijin haohuaijin commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The push decoder evaluates each ArrowPredicate separately. Consecutive predicates with the same projection may therefore decode the same column repeatedly or replay it through the predicate cache.

This PR evaluates eligible same-projection predicates from one decoded stream. Later predicates receive only rows accepted by earlier predicates. Intermediate results use pass-through, zero-copy slice, compaction, or early stop as appropriate.

Fusion is limited to a single top-level, non-repeated leaf projection and is disabled when a row limit is active. Other predicate shapes continue to use the existing path. The synchronous reader is unchanged.

Performance

Fusion trades repeated decoding or predicate-cache replay for survivor materialization. It can improve selective or expensive predicates, but may regress inexpensive predicates when most rows survive or survivors are fragmented.

ClickBench Q25 latency decreased from 140.379 ms to 110.727 ms (-21.12%). Across 256 three-predicate cases, geometric-mean latency decreased by 6.84%, although 33 cases regressed by more than 2%. The worst regressions occurred with early 99%-survivor predicates.

Full benchmark results: #10926 (comment)

What changes are included?

  • Detect eligible consecutive same-projection predicates in the push decoder
  • Evaluate each group from one decoded stream
  • Preserve predicate order and RowSelection semantics
  • Keep unsupported cases and the synchronous reader on the existing path

Testing

  • Fused and sequential RowSelection equivalence, with and without a prior selection
  • Materialization classification and fusion eligibility
  • Push-decoder end-to-end coverage
  • Existing Parquet library tests

User-facing changes

None. This is an internal RowFilter optimization.

@haohuaijin

haohuaijin commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant