Skip to content

HIVE-29829: Iceberg: Partition-level column statistics refactor - #6707

Open
deniskuzZ wants to merge 5 commits into
apache:masterfrom
deniskuzZ:col_stats_refactor
Open

HIVE-29829: Iceberg: Partition-level column statistics refactor#6707
deniskuzZ wants to merge 5 commits into
apache:masterfrom
deniskuzZ:col_stats_refactor

Conversation

@deniskuzZ

@deniskuzZ deniskuzZ commented Aug 18, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

@deniskuzZ
deniskuzZ marked this pull request as draft August 18, 2026 17:14
@deniskuzZ
deniskuzZ force-pushed the col_stats_refactor branch 2 times, most recently from f73473a to 04270e5 Compare August 19, 2026 20:52
@deniskuzZ deniskuzZ changed the title WIP: Iceberg col stats refactor HIVE-29829: Iceberg: Partition-level column statistics refactor Aug 21, 2026
@deniskuzZ
deniskuzZ marked this pull request as ready for review August 21, 2026 09:08
@deniskuzZ
deniskuzZ requested a lite review from Copilot August 21, 2026 09:16
Copilot AI reviewed Aug 21, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors Hive/Iceberg column-statistics handling to better support partition-level semantics and snapshot/branch scenarios, including introducing a reader-materialized partition name virtual column.

Changes:

  • Add PARTITION__NAME virtual column and propagate it through vectorized and row-mode read paths for non-native partition support.
  • Rework column stats collection/persistence for non-native tables (Iceberg), including branch/snapshot-aware behavior and a new stats write policy.
  • Update planner/optimizer and a large set of Iceberg query result baselines to reflect the new virtual column and revised stats behavior.

Reviewed changes

Copilot reviewed 92 out of 99 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java Serve table-level col stats over pruned partitions for non-native tables; refine data-size derivation.
ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java Adjust partition name derivation and storage-handler-based stats persistence behavior.
ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java Skip HMS partition handling for non-native partition support; adjust partition list retrieval.
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java Rewrite ANALYZE for non-native partition support using PARTITION__NAME grouping.
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java Disable partition-stats path for non-native partition support in auto-gather pipeline.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java Switch vectorization virtual column from partition projection to partition name.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java Preserve snapshot ref when constructing stats work from file sink context.
ql/src/java/org/apache/hadoop/hive/ql/metadata/VirtualColumn.java Rename virtual column to PARTITION_NAME and update registry.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java Adjust validity/virtual-column exposure for non-native partition support and meta tables.
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java Remove unused partition transform specs API.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java Prevent HMS col-stats reads for non-native handler/branch snapshot scenarios.
ql/src/java/org/apache/hadoop/hive/ql/io/PositionDeleteInfo.java Remove partition-projection from delete context payload.
ql/src/java/org/apache/hadoop/hive/ql/io/PartitionInfo.java New IOContext payload for reader-materialized partition name.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java Use row batch context column type infos when reading batches.
ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java Track and parse PartitionInfo alongside other IO context info.
ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java Parse PartitionInfo on each next() for non-ACID readers.
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java Restore snapshot ref on table lookups for stats tasks.
ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java Populate PARTITION__NAME virtual column from IOContext PartitionInfo.
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java Prefer table-level stats for non-native tables that don’t maintain partition stats.
iceberg/iceberg-handler/src/test/results/positive/vectorized_iceberg_read_parquet.q.out Update expected plans/output for PARTITION__NAME virtual column.
iceberg/iceberg-handler/src/test/results/positive/vectorized_iceberg_read_orc.q.out Update expected plans/output for PARTITION__NAME virtual column.
iceberg/iceberg-handler/src/test/results/positive/vectorized_iceberg_read_mixed.q.out Update expected plans/output for PARTITION__NAME virtual column.
iceberg/iceberg-handler/src/test/results/positive/update_iceberg_partitioned_avro.q.out Refresh expected plan warnings/order changes.
iceberg/iceberg-handler/src/test/results/positive/row_count.q.out Update expected stats/plan output reflecting revised stats pipeline.
iceberg/iceberg-handler/src/test/results/positive/query_iceberg_metadata_of_partitioned_table.q.out Adjust expected output for metadata queries after refactor.
iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc8.q.out Update expected plan output for removed/changed virtual column projection.
iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc7.q.out Update expected plan output for removed/changed virtual column projection.
iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc5.q.out Update expected plan output for removed/changed virtual column projection.
iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc4.q.out Update expected plan output for removed/changed virtual column projection.
iceberg/iceberg-handler/src/test/results/positive/metadata_delete.q.out Update expected plan output after virtual column changes.
iceberg/iceberg-handler/src/test/results/positive/merge_iceberg_orc.q.out Update expected plan output after virtual column changes.
iceberg/iceberg-handler/src/test/results/positive/llap/vectorized_iceberg_read_parquet.q.out Update expected LLAP vectorization schema for PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/llap/vectorized_iceberg_read_orc.q.out Update expected LLAP vectorization schema for PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/llap/vectorized_iceberg_read_mixed.q.out Update expected LLAP vectorization schema for PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/llap/llap_iceberg_read_orc.q.out Refresh expected LLAP plan/stats output after stats changes.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_minor_compaction_unpartitioned.q.out Update expected COLUMN_STATS_ACCURATE output after stats persistence change.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_minor_compaction_partition_evolution.q.out Update expected COLUMN_STATS_ACCURATE output after stats persistence change.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_minor_compaction_bucket.q.out Update expected COLUMN_STATS_ACCURATE output after stats persistence change.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_major_compaction_single_partition_with_evolution2.q.out Update expected COLUMN_STATS_ACCURATE output after stats persistence change.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_major_compaction_partition_evolution2.q.out Update expected COLUMN_STATS_ACCURATE output after stats persistence change.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_major_compaction_partition_evolution.q.out Refresh expected plan/stats output after compaction/stats refactor.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_major_compaction_partition_evolution_ordered.q.out Update expected COLUMN_STATS_ACCURATE output after stats persistence change.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_create_locally_zordered_table.q.out Update expected plan edges/aggregations after stats pipeline changes.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_8.q.out Update expected plan/stats output after stats changes.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_6.q.out Update expected plan/stats output after stats changes.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_5.q.out Update expected plan/stats output after stats changes.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_4.q.out Update expected plan/stats output after stats changes.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_3.q.out Update expected plan/stats output after stats changes.
iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_2.q.out Update expected plan/stats output after stats changes.
iceberg/iceberg-handler/src/test/results/positive/llap/hadoop_catalog_create_table.q.out Update expected COLUMN_STATS_ACCURATE output after stats persistence change.
iceberg/iceberg-handler/src/test/results/positive/iceberg_truncate_partition_with_evolution.q.out Update expected stats/plan output after virtual column/stats refactor.
iceberg/iceberg-handler/src/test/results/positive/iceberg_stats_with_ppr.q.out Update expected column stats state after refactor.
iceberg/iceberg-handler/src/test/results/positive/iceberg_pcr_null_partition.q.out Update expected column stats state after refactor.
iceberg/iceberg-handler/src/test/results/positive/iceberg_merge_schema.q.out Update expected vectorization schema output for PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/iceberg_load_data.q.out Update expected plan edges/shuffles after stats changes.
iceberg/iceberg-handler/src/test/results/positive/iceberg_insert_into_partition_with_evolution.q.out Update expected plan edges/output after stats changes.
iceberg/iceberg-handler/src/test/results/positive/iceberg_explain_formatted.q.out Update expected formatted explain JSON for PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/dynamic_semijoin_reduction.q.out Update expected plan/stats output after stats changes.
iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_mixed.q.out Update expected plan/stats output after delete path changes.
iceberg/iceberg-handler/src/test/results/positive/ctas_iceberg_partitioned_orc.q.out Update expected COLUMN_STATS_ACCURATE output after stats persistence change.
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out Update expected COLUMN_STATS_ACCURATE per-column map after refactor.
iceberg/iceberg-handler/src/test/results/positive/bucket_map_join_9.q.out Update expected plan/stats output after stats changes.
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/writer/TestHiveIcebergDeleteWriter.java Update delete-record schema creation for new delete serde schema signature.
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/vector/TestHiveIcebergVectorization.java Pass FileScanTask into batch iterator to support task-scoped virtuals.
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergSelects.java Add test ensuring PARTITION__NAME predicates are not pushed down to Iceberg.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergRecordReader.java Make virtual column iterator task-aware and simplify constants map logic.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergMergeRecordReader.java Use merge-task delete schema and pass spec/partition info for virtuals.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/AbstractIcebergRecordReader.java Update virtual-column schema construction to new helper signature.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/HiveIcebergDeleteWriter.java Use merge-aware position delete extraction.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/HiveIcebergCopyOnWriteRecordWriter.java Use updated position delete extraction signature.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveVectorizedReader.java Thread FileScanTask through batch iterator creation.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveDeleteFilter.java Use diamond operator for CloseableIterable implementation.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveBatchIterator.java Populate virtual columns from FileScanTask, including PARTITION__NAME.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/stats/ColStatsWritePolicy.java New policy enum to decide replace/merge/skip for stats file writes.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergAcidUtil.java Rework virtual column plumbing and delete serde schemas; add PartitionInfo propagation.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergSerDe.java Update projected schema logic and merge-task delete schema selection.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergCompactionService.java Disable follow-up compute stats for compaction since stats are gathered in-query.
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/MetastoreUtil.java Simplify partition key derivation (currently still disabled via TODO).
Suppressed comments (1)

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveBatchIterator.java:136

  • BytesColumnVector.fill does not reset noNulls, and String#getBytes() is platform-default charset. Since PARTITION_NAME is always set for Iceberg tasks (uses DummyPartition.VOID for unpartitioned), set noNulls=true and encode with UTF-8 for deterministic behavior and less null-handling overhead.
            case PARTITION_NAME:
              bcv = (BytesColumnVector) batch.cols[idx];
              bcv.fill(partitionName.getBytes());
              break;

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveBatchIterator.java
Comment on lines 121 to 124
case FILE_PATH:
value = idToConstant.get(MetadataColumns.FILE_PATH.fieldId());
BytesColumnVector bcv = (BytesColumnVector) batch.cols[idx];
if (value == null) {
bcv.noNulls = false;
bcv.isNull[0] = true;
bcv.isRepeating = true;
} else {
bcv.fill(((String) value).getBytes());
}
bcv.fill(filePath.getBytes());
break;
@deniskuzZ
HIVE-29781: Introduce PARTITION__NAME virtual column and refactor Ice…
dbf09be 
…berg virtual column plumbing

Adds VirtualColumn.PARTITION_NAME, materialized by the Iceberg readers: vectorized via
HiveBatchIterator, row mode via PartitionInfo -> IOContext -> MapOperator.

Refactors the read path to take everything the file already carries from the scan task
instead of the per-record constants map: spec id, partition hash, file path, partition
name, first row id and file sequence number are computed once per task in both
VirtualColumnAwareIterator and HiveBatchIterator. Drops the now unread _partition column
from the ACID read schema, along with the write-only VIRTUAL_COLS_TO_META_COLS map and the
per-record helpers it fed.

Fixes VectorizedParquetRecordReader taking a column type from the job column list while
indexing it with a batch column number: the two orders diverge past the data columns
because the batch drops virtual columns the readers cannot fill.
@deniskuzZ
HIVE-29781: Make PARTITION__PROJECTION a merge task record field inst…
2f642c5 
…ead of a virtual column

The partition key a merge task carries is read back only by HiveIcebergDeleteWriter under
isMergeTask: an ordinary delete derives the key from the row data, and a merge task cannot
because it reads delete files, whose records hold only a path and a position.

Declaring it as a virtual column put an always empty string into every DELETE, UPDATE and
MERGE record and into every Iceberg plan. It now lives in a merge specific serde layout that
IcebergMergeRecordReader and HiveIcebergSerDe select on the merge task flag, so the ordinary
delete record is one column shorter and the plans no longer estimate it.
@deniskuzZ
HIVE-29829: Iceberg partition-level column statistics refactor
51824c0
@deniskuzZ
HIVE-29829: Iceberg column statistics granularity and branch-scoped s…
df23971 
…tatistics

Adds hive.iceberg.stats.collect.partlevel (default false), scoping column
  statistics granularity for Iceberg tables independently of the generic
  hive.analyze.stmt.collect.partlevel.stats. Partition-level statistics are
  produced by ANALYZE; every other write maintains table-level ones.

Keys the column-statistics read path on the calling API rather than on session
  config, so a granularity mismatch cannot surface a wrongly-shaped blob.

Stores branch-scoped statistics on the branch's snapshot and stops branch
  writes from overwriting the table-scoped metastore row, including the footer
  scan path used by ANALYZE ... COMPUTE STATISTICS.

Lets a whole-table INSERT OVERWRITE of a partitioned table replace its column
  statistics, which previously skipped because whole-table and partition-scoped
  overwrites are indistinguishable in the snapshot summary.
@deniskuzZ
HIVE-29829: Iceberg write-path partition column statistics
2419328 
  Derive partition names through the Iceberg spec so a write keys statistics
  the way a read looks them up; substitute per partition for a scoped write
  or ANALYZE; gather partition-level statistics for CTAS.
@sonarqubecloud

sonarqubecloud Bot commented Aug 22, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
23 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.7% Duplication on New Code

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

tests pending

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@deniskuzZ @asf-ci-hive