[WIP] Let StatsOptimizer answer from a storage handler's column statistics - #6716
Open
deniskuzZ wants to merge 5 commits into
Open
[WIP] Let StatsOptimizer answer from a storage handler's column statistics#6716deniskuzZ wants to merge 5 commits into
deniskuzZ wants to merge 5 commits into
Conversation
…berg virtual column plumbing Adds VirtualColumn.PARTITION_NAME, materialized by the Iceberg readers: vectorized via HiveBatchIterator, row mode via PartitionInfo -> IOContext -> MapOperator. Refactors the read path to take everything the file already carries from the scan task instead of the per-record constants map: spec id, partition hash, file path, partition name, first row id and file sequence number are computed once per task in both VirtualColumnAwareIterator and HiveBatchIterator. Drops the now unread _partition column from the ACID read schema, along with the write-only VIRTUAL_COLS_TO_META_COLS map and the per-record helpers it fed. Fixes VectorizedParquetRecordReader taking a column type from the job column list while indexing it with a batch column number: the two orders diverge past the data columns because the batch drops virtual columns the readers cannot fill.
…ead of a virtual column The partition key a merge task carries is read back only by HiveIcebergDeleteWriter under isMergeTask: an ordinary delete derives the key from the row data, and a merge task cannot because it reads delete files, whose records hold only a path and a position. Declaring it as a virtual column put an always empty string into every DELETE, UPDATE and MERGE record and into every Iceberg plan. It now lives in a merge specific serde layout that IcebergMergeRecordReader and HiveIcebergSerDe select on the merge task flag, so the ordinary delete record is one column shorter and the plans no longer estimate it.
…tatistics Adds hive.iceberg.stats.collect.partlevel (default false), scoping column statistics granularity for Iceberg tables independently of the generic hive.analyze.stmt.collect.partlevel.stats. Partition-level statistics are produced by ANALYZE; every other write maintains table-level ones. Keys the column-statistics read path on the calling API rather than on session config, so a granularity mismatch cannot surface a wrongly-shaped blob. Stores branch-scoped statistics on the branch's snapshot and stops branch writes from overwriting the table-scoped metastore row, including the footer scan path used by ANALYZE ... COMPUTE STATISTICS. Lets a whole-table INSERT OVERWRITE of a partitioned table replace its column statistics, which previously skipped because whole-table and partition-scoped overwrites are indistinguishable in the snapshot summary.
… statistics StatsOptimizer fetched partition column statistics straight from the metastore, which cannot hold them for an Iceberg table: PART_COL_STATS rows require a PARTITIONS row that Iceberg never creates. Aggregates over a partitioned Iceberg table therefore always executed, even with statistics available in Puffin files. Fetch through Hive.getAggrColStatsFor, which consults the storage handler, mirroring what getRowCnt already does for row counts. The statistics arrive aggregated over the pruned partitions rather than one entry per partition; the callers folded them with min, max or a sum, so merging first gives the same answer and the folds collapse. getPartsFound() replaces the previous size check, so a partition without statistics still stops the optimization. Accuracy is asked of the component that owns it. The metastore's marker only records Hive's own writes, while an Iceberg table may be written by other engines, so areColumnStatsUptoDate is added to HiveStorageHandler and answered from table metadata by the Iceberg handler. The default keeps the metastore marker and the canProvideBasicStatistics guard, so other handlers are unchanged, and the Iceberg override defers to the metastore when hive.iceberg.stats.source says the statistics live there.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?