fix: ignore reader-side parquet.hadoop.vectored.io.enabled in Iceberg native-write detection - #5410
Merged
kazuyukitanimura merged 2 commits intoAug 21, 2026
Conversation
… native-write detection Avoid mis-classifying Compatible Iceberg V2 writes as Unsupported due to well-known parquet reader settings. CometIcebergNativeWrite's `requireNoParquetHadoopConfOverrides` (added in apache#5298) walks the entire session Hadoop Configuration and returns an `Unsupported` fall-back reason for any key starting with `parquet.`. The intent is sound (a `parquet.*` key in the Hadoop conf reaches iceberg-java's writer but not Comet's native writer), but the predicate is too broad: it checks reader-side `parquet.*` keys the same as writer-side ones. `parquet.hadoop.vectored.io.enabled` is a reader-side vectored-IO knob: - Declared in parquet-hadoop as `org.apache.parquet.hadoop.ParquetInputFormat.HADOOP_VECTORED_IO_ENABLED` with default `true` in parquet-hadoop 1.16+. - Only consulted by parquet-mr's Hadoop reader path (via `HadoopReadOptions.useHadoopVectoredIo`); iceberg-java's writer never reads it. Any environment that seeds it into the session Hadoop configuration trips this rule, and every otherwise-Compatible Iceberg V2 write is silently mis-classified as Unsupported — the native writer is disabled without the user having changed any writer-relevant setting. Add an `IgnoredHadoopParquetConfKeys` set containing `parquet.hadoop.vectored.io.enabled` and consult it inside the `parquet.*` walk before emitting the fall-back reason. The broad `startsWith("parquet.")` gate remains in place for every writer-relevant knob. Add a regression test that pins `parquet.hadoop.vectored.io.enabled=true` in the session Hadoop configuration via `withSQLConf` and asserts Compatible. The existing negative test ("fall-back: parquet.* key in the session Hadoop configuration", keyed on `parquet.block.size`) continues to exercise the rejection path for real writer-relevant `parquet.*` keys.
kazuyukitanimura
approved these changes
Aug 21, 2026
kazuyukitanimura
left a comment
kazuyukitanimura
left a comment
Contributor
There was a problem hiding this comment.
Thanks
pending ci
Contributor
|
Thanks merged @snmvaughan |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CometIcebergNativeWrite's
requireNoParquetHadoopConfOverrides(added in #5298) walks the entire session Hadoop Configuration and returns anUnsupportedfall-back reason for any key starting withparquet.. The intent is sound (aparquet.*key in the Hadoop conf reaches iceberg-java's writer but not Comet's native writer), but the predicate is too broad: it checks reader-sideparquet.*keys the same as writer-side ones.parquet.hadoop.vectored.io.enabledis a reader-side vectored-IO knob:org.apache.parquet.hadoop.ParquetInputFormat.HADOOP_VECTORED_IO_ENABLEDwith defaulttruein parquet-hadoop 1.16+.HadoopReadOptions.useHadoopVectoredIo); iceberg-java's writer never reads it.Any environment that seeds it into the session Hadoop configuration trips this rule, and every otherwise-Compatible Iceberg V2 write is silently mis-classified as Unsupported — the native writer is disabled without the user having changed any writer-relevant setting.
Which issue does this PR close?
Closes #.
Rationale for this change
Avoid mis-classifying Compatible Iceberg V2 writes as Unsupported due to well-known parquet reader settings.
What changes are included in this PR?
Add an
IgnoredHadoopParquetConfKeysset containingparquet.hadoop.vectored.io.enabledand consult it inside theparquet.*walk before emitting the fall-back reason. The broadstartsWith("parquet.")gate remains in place for every writer-relevant knob.How are these changes tested?
Add a regression test that pins
parquet.hadoop.vectored.io.enabled=truein the session Hadoop configuration viawithSQLConfand asserts Compatible. The existing negative test ("fall-back: parquet.* key in the session Hadoop configuration", keyed onparquet.block.size) continues to exercise the rejection path for real writer-relevantparquet.*keys.