HIVE-29828: Introduce PARTITION__NAME virtual column and refactor Iceberg virtual column plumbing

[deniskuzZ]

Adds VirtualColumn.PARTITION_NAME, materialized by the Iceberg readers: vectorized via HiveBatchIterator, row mode via PartitionInfo -> IOContext -> MapOperator.

Refactors the read path to take everything the file already carries from the scan task instead of the per-record constants map: spec id, partition hash, file path, partition name, first row id and file sequence number are computed once per task in both VirtualColumnAwareIterator and HiveBatchIterator. Drops the now unread _partition column from the ACID read schema, along with the write-only VIRTUAL_COLS_TO_META_COLS map and the per-record helpers it fed.

Fixes VectorizedParquetRecordReader taking a column type from the job column list while indexing it with a batch column number: the two orders diverge past the data columns because the batch drops virtual columns the readers cannot fill.

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

[Copilot]

Pull request overview

Introduces a new PARTITION__NAME virtual column and refactors Iceberg virtual-column materialization so task-level constants (spec id, partition hash, file path, partition name, row lineage offsets/sequence) are computed once per scan task for both row-mode and vectorized reads. The PR also removes the legacy PARTITION__PROJECTION virtual column usage and fixes a type-indexing issue in VectorizedParquetRecordReader where job column order diverges from batch column order once virtual columns are involved.

Changes:

• Add VirtualColumn.PARTITION_NAME (PARTITION__NAME) and thread it through Hive/Iceberg read paths (row-mode via PartitionInfo -> IOContext -> MapOperator, vectorized via HiveBatchIterator).
• Refactor Iceberg readers/iterators to pass FileScanTask (and derived constants) instead of a per-record constants map; adjust delete/merge writer schemas accordingly.
• Fix vectorized Parquet type lookup to use batch-column indexing (rbCtx.getRowColumnTypeInfos()), preventing mismatches when virtual columns are present.

Reviewed changes

Copilot reviewed 46 out of 46 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java	Switch vectorizable virtual column from PARTITION__PROJECTION to PARTITION__NAME.
ql/src/java/org/apache/hadoop/hive/ql/metadata/VirtualColumn.java	Add PARTITION_NAME virtual column and update virtual column registries/maps.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java	Adjust which virtual columns are exposed for non-native tables, including PARTITION_NAME when supported.
ql/src/java/org/apache/hadoop/hive/ql/io/PositionDeleteInfo.java	Remove persisted partitionProjection from position-delete context.
ql/src/java/org/apache/hadoop/hive/ql/io/PartitionInfo.java	New per-reader partition context container for PARTITION__NAME.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java	Fix column type lookup to align with batch column numbering.
ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java	Add PartitionInfo to IOContext and parsing hook.
ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java	Parse PartitionInfo into IOContext on each record (row mode).
ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java	Populate PARTITION__NAME from IOContext.PartitionInfo.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergRecordReader.java	Pass FileScanTask into row-mode virtual-column iterator; simplify constants map creation.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergMergeRecordReader.java	Update delete schema creation for merge tasks and adjust merge-task VC iterator inputs.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/AbstractIcebergRecordReader.java	Update file read schema creation call signature.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveVectorizedReader.java	Pass FileScanTask into HiveBatchIterator; modernize anonymous generic instantiation.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveDeleteFilter.java	Modernize anonymous generic instantiation.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveBatchIterator.java	Materialize Iceberg virtual columns (including PARTITION__NAME) from FileScanTask once per task.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java	Provide public toPartitionName(PartitionSpec, StructLike) utility.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergAcidUtil.java	Refactor Iceberg virtual-column plumbing; adjust SerDe/read schemas and iterators to use task-derived constants and PartitionInfo.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java	Remove legacy PARTITION_PROJECTION from Iceberg ACID virtual columns list.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergSerDe.java	Update schema projection logic and delete-schema selection to account for merge-task differences.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/HiveIcebergDeleteWriter.java	Update position-delete extraction for merge vs non-merge tasks.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/HiveIcebergCopyOnWriteRecordWriter.java	Update position-delete extraction call signature.
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/writer/TestHiveIcebergDeleteWriter.java	Update test record schema construction for delete operations.
iceberg/iceberg-handler/src/test/results/positive/write_iceberg_branch.q.out	Update expected plan/output after removing PARTITION__PROJECTION and column reindexing.
iceberg/iceberg-handler/src/test/results/positive/vectorized_iceberg_read_parquet.q.out	Update expected outputs to reference PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/vectorized_iceberg_read_orc.q.out	Update expected outputs to reference PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/vectorized_iceberg_read_mixed.q.out	Update expected outputs to reference PARTITION__NAME and updated column ordering.
iceberg/iceberg-handler/src/test/results/positive/vectorized_iceberg_merge_mixed.q.out	Update expected outputs to reflect virtual-column/schema changes and column numbering shifts.
iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc7.q.out	Update expected outputs after virtual column changes.
iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc5.q.out	Update expected outputs after virtual column changes.
iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_orc4.q.out	Update expected outputs after virtual column changes.
iceberg/iceberg-handler/src/test/results/positive/metadata_delete.q.out	Update expected outputs after virtual column changes.
iceberg/iceberg-handler/src/test/results/positive/merge_iceberg_partitioned_orc.q.out	Update expected outputs after virtual column changes.
iceberg/iceberg-handler/src/test/results/positive/merge_iceberg_orc.q.out	Update expected outputs after virtual column changes.
iceberg/iceberg-handler/src/test/results/positive/merge_iceberg_copy_on_write_unpartitioned.q.out	Update expected outputs after virtual column changes.
iceberg/iceberg-handler/src/test/results/positive/merge_iceberg_copy_on_write_partitioned.q.out	Update expected outputs after virtual column changes.
iceberg/iceberg-handler/src/test/results/positive/llap/vectorized_iceberg_read_parquet.q.out	Update expected LLAP outputs to reference PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/llap/vectorized_iceberg_read_orc.q.out	Update expected LLAP outputs to reference PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/llap/vectorized_iceberg_read_mixed.q.out	Update expected LLAP outputs to reference PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/iceberg_truncate_partition_with_evolution.q.out	Update expected outputs after virtual column removal/reindexing.
iceberg/iceberg-handler/src/test/results/positive/iceberg_merge_schema.q.out	Update expected outputs to reference PARTITION__NAME and related projection changes.
iceberg/iceberg-handler/src/test/results/positive/iceberg_explain_formatted.q.out	Update explain formatted output to show PARTITION__NAME.
iceberg/iceberg-handler/src/test/results/positive/delete_iceberg_mixed.q.out	Update expected outputs after virtual column changes and column reindexing.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[deniskuzZ]

wrong, FILE_READ_META_COLS.size() stands for the same offset

[deniskuzZ]

PARTITION__PROJECTION intended for internal use by merge task only

[Copilot]

Pull request overview

Copilot reviewed 50 out of 50 changed files in this pull request and generated no new comments.

Suppressed comments (1)

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java:500

• toPartitionName calls spec.partitionToPath(data) without guarding against data == null. The rest of the PR treats the partition struct as nullable (e.g., computeHash returns -1 when task.file().partition() is null), and both HiveBatchIterator and VirtualColumnAwareIterator now call toPartitionName(task.spec(), task.file().partition()) directly. If Iceberg supplies null for unpartitioned files, this will NPE when materializing PARTITION__NAME.

  public static String toPartitionName(PartitionSpec spec, StructLike data) {
    String path = spec.partitionToPath(data);
    // an unpartitioned spec renders nothing: its rows belong to the table-level partition
    return path.isEmpty() ? DummyPartition.VOID : path;

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.0% Duplication on New Code

See analysis details on SonarQube Cloud