Antalya 26.6 - Backport of #115603 - read row_lineage(iceberg v3) fields as virtual columns

[subkanthi]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Iceberg v3 has two new fields in manifest file - first_row_id and last_seq_num. This PR allows to read them as virtual columns from system tables(system.iceberg_files) (ClickHouse#115603 by @scanhex12)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [868dfb9]

[subkanthi]

@blau-ai

[blau-ai]

CI triage for #2308 (Iceberg v3 row_lineage backport)

Verdict: 7 failing checks — 0 caused by this PR. All are pre-existing base-branch failures, flaky/infra, or unrelated test-maintenance. The PR's own build & tests are green (all Builds, Fast test, Unit tests, gtest_iceberg_count_shortcuts, and stateless 04203_system_iceberg_files all pass; the added test_row_lineage*.py are not among the failures).

The strongest single piece of evidence: I compared against #2309 ("Bump to 26.6.4"), a trivial version bump that contains none of this Iceberg code. It reproduces the same integration, cas‑s3 stateless, and settings regression failures — so those are properties of the antalya-26.6 base/CI environment, not of this diff. The diff itself is entirely under src/Storages/ObjectStorage/DataLakes/Iceberg/*, ProtocolDefines.h, VirtualColumnUtils, and StorageObjectStorageSource — none of which the failing suites exercise.

---

1. Integration tests (amd_asan_ubsan, db disk, old analyzer, 4/8) — 12/1102 ❌ (blocking) → pre-existing, not PR-caused

All 12 failures are test_storage_iceberg_with_spark/test_schema_inference.py::test_schema_inference[...] (the s3/azure/local × 1/2 × True/False matrix). The first case crashes the server on SELECT * FROM <iceberg table> with ATTEMPT_TO_READ_AFTER_EOF (stack: throwReadAfterEOF → Connection::receivePacket); the remaining 11 then fail with Connection refused (…:9000) NETWORK_ERROR because the server is already dead.

• test_schema_inference.py is a pre-existing test (this PR only added test_row_lineage.py / test_row_lineage_pruning.py).
• Antalya 26.6: Bump to 26.6.4 #2309 fails the exact same 12 tests with the identical crash-then-connection-refused signature (only the version string 26.6.4 vs 26.6.2 and the minio IP differ). → This is a base-branch / spark-iceberg test-env instability on antalya-26.6, reproduced by any PR retargeted onto it.
• Next step: track separately as a base-branch flaky/crash (worth a server-log dig by whoever owns the iceberg-spark integration env); re-running won't clear it, but it is not a regression from this backport.

2. Stress test (amd_debug) — 1/5 ❌ (blocking) → not PR-caused (flaky/base instability)

Single failure: Lost connection to server (server crash, no stack in report; fatal.log empty). Stress test (amd_asan_ubsan) passed 0/4, and Stress (amd_debug) passes on #2309. No evidence tying it to the Iceberg diff.

• Next step: safe to re-run; if it recurs, pull clickhouse-server.err.log from the stress job to confirm it's the same base-branch crash.

3. Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2) — Failed: 1 ❌ (non-blocking, do_not_block_pipeline_on_failure) → infra, not PR-caused

Failure is Connection refused (e.code=111) to the cas‑s3 backend during a plain CREATE TABLE … ENGINE=MergeTree (non-Iceberg test), which then took the server down. Flaky cas‑s3 storage backend; suite is explicitly non-blocking.

4. Stateless tests (amd_binary, cas s3 storage, parallel) — Failed: 2 ❌ (non-blocking) → flaky, not PR-caused

Both CI-labeled flaky:

• 02265_column_ttl — NETWORK_ERROR on a ReplicatedMergeTree part relink (13/15 same-settings reruns failed → flagged flaky by CI).
• 03629_duplicate_partition_keys_crash — S3 Connection refused to the cas_s3 bucket; all 6 reruns passed (transient).
  Neither touches Iceberg. Antalya 26.6: Bump to 26.6.4 #2309 shows the same cas‑s3 suites red with a shifting set of tests → environmental.

5. RegressionTestsRelease / CAS (selects) / cas_selects ❌ → effectively passed / flaky

The /selects module is OK (~99.9%); the only non-OK rows are 2 known XFails (/final/modifier "not implemented", "select union clause with alias" "need to add retry"). Zero real fails; the red status is a first-attempt/retry artifact. Passes on #2309.

6. RegressionTestsRelease / Common (settings) / settings ❌ → not PR-caused (snapshot maintenance)

SnapshotNotFoundError in default_values.py: the regression snapshot has no entry for a new server setting export_merge_tree_part_ignore_extra_source_columns (actual {"default":"0"}). That setting comes from the separate export … partition_by_name work, not from this Iceberg backport (this PR adds no server settings — it only bumps protocol-version constants in ProtocolDefines.h). Also fails on #2309.

• Next step (separate task, not this PR): regenerate the settings regression snapshot to include the new setting.

7. Regression release tiered_storage_minio ❌ (module errored) → infra teardown timeout

ExpectTimeoutError: Timeout 1000.000s in helpers/cluster.py __exit__ — the docker-compose cluster teardown hung after ~1h57m. All 90 executed scenarios passed (4 XFail). Passes on #2309.

• Next step: safe to re-run.

---

Recommendation

No code change to this PR is warranted — nothing in the red CI is attributable to the row_lineage diff, and the PR's own tests pass. The settings regression snapshot regen and the pre-existing test_schema_inference iceberg-spark crash on antalya-26.6 are real but belong to separate base-branch/test-maintenance tracks. The flaky/infra suites (stress, cas‑s3 stateless, cas_selects, tiered_storage_minio) are safe to re-run.

Analysis is evidence-first from the praktika S3 reports and a control comparison with #2309; I cannot build or run ClickHouse in this environment. Head SHA c9a0c51.

[subkanthi]

TESTING
Altinity/ice#214

ice insert --format-version=3 -p flowers.irisv3 file://iris.parquet

CH

 SELECT first_row_id,sequence_number FROM system.iceberg_files where database=currentDatabase() and table='flowers.irisv3';

SELECT
    first_row_id,
    sequence_number
FROM system.iceberg_files
WHERE (database = currentDatabase()) AND (`table` = 'flowers.irisv3')

Query id: c0012f98-d809-4a40-a3bc-799284d4a19a

   ┌─first_row_id─┬─sequence_number─┐
1. │            0 │               1 │
   └──────────────┴─────────────────┘

1 row in set. Elapsed: 0.075 sec. 

After inserting 150 rows, ice does Append

insert --format-version=3 -p flowers.irisv3 file://iris.parquet

SELECT first_row_id,sequence_number FROM system.iceberg_files where database=currentDatabase() and table='flowers.irisv3';

SELECT
    first_row_id,
    sequence_number
FROM system.iceberg_files
WHERE (database = currentDatabase()) AND (`table` = 'flowers.irisv3')

Query id: 8e405717-c67b-4944-945c-dbff2240ddf0

   ┌─first_row_id─┬─sequence_number─┐
1. │          150 │               2 │
2. │            0 │               1 │
   └──────────────┴─────────────────┘

[ianton-ru]

Code the same as in original PR, but I recommend also to keep the same format (indents, empty lines, brackets) to avoid additional conflicts in future. I did not mark all places with different formatting.

[ianton-ru]

This block a little bit reformatted (removed empty lines, brackets). This does not change something, but can create additional conflicts in future backports instead of automatic solving during git merge/git cherry-pick.

[ianton-ru]

The same as above - block is reformatted (different indent).

[ianton-ru]

Also - different indents.

[subkanthi]

@blau-ai

[blau-ai]

Triage of CI on ae6c492 (latest commit): 0 failures — no red checks. CI is not fully green yet only because the run is still in progress; it was kicked off ~11 min ago by the reformat indentation/spaces commit.

Status snapshot

State	What	Notes
✅ SUCCESS	Config Workflow, CI Tests (0/155), DCO	all green
🟡 RUNNING	Fast test, Source upload	in progress
⏳ PENDING	Builds (amd), Stateless, Integration, Unit, Stress, AST/BuzzHouse fuzzers, SQLLogic/SQLStorm	not started yet
⏭️ SKIPPED	all arm / tsan / msan variants, Keeper stress, toolchain	excluded by CI tags, not failures

Nothing is in a FAILED/ERROR state — I checked the full praktika report (result_pr.json), not just the GitHub check summaries. So there are no PR-related failures to fix, and nothing pre-existing/flaky/infra has surfaced either.

What still needs to run — the jobs that actually exercise this backport haven't executed yet on this commit:

• Integration tests (amd_asan_ubsan) — will run the new test_row_lineage.py / test_row_lineage_pruning.py.
• Unit tests (asan_ubsan) — will run the updated gtest_iceberg_count_shortcuts.
• Stateless tests — will check the updated 04203_system_iceberg_files.reference.
• Builds (amd_debug / asan_ubsan / binary / release) — will confirm the C++ changes compile (they touch ProtocolDefines.h, IcebergMetadata, AvroForIcebergDeserializer, VirtualColumnUtils, StorageSystemIcebergFiles, etc.).

Note: an earlier run on c9a0c51 never completed — it was superseded when the two antalya-26.6 merge commits and the reformat commit were pushed — so there is no fully-green prior run to point to. The real verdict comes from the run now in flight.

Health check

This is a well-scoped backport of upstream ClickHouse#115603 (read row_lineage / first_row_id / last_seq_num from Iceberg v3 manifests as virtual columns): 24 files, contained to the Iceberg datalake code plus a protocol-version bump, and it ships its own coverage — two integration tests, a gtest update, and a stateless reference update. The latest commit is whitespace-only alignment to upstream, and it's cleanly merged up to antalya-26.6. No structural red flags.

Next step: let the run finish, then re-run @blau-ai (or ping me) once the Build / Integration / Unit / Stateless jobs report and I'll classify anything that turns red. Right now there's nothing actionable.

_{— automated CI triage by @blau-ai}

[ianton-ru]

@blau-ai

[blau-ai]

CI triage for #2308 (backport of ClickHouse#115603 — Iceberg v3 first_row_id / _row_id virtual columns)

Verdict: 0 of the failing checks are caused by this PR. Every red check is CI infrastructure (a network outage during the run), a session timeout, or a flaky test in a feature area this PR does not touch. No crashes, no sanitizer hits, no compile errors, and nothing Iceberg-related in any failure log.

The head commit 868dfb9 changes only Iceberg data‑lake C++ (AvroForIcebergDeserializer, IcebergDataObjectInfo, IcebergMetadata, ManifestFile*, Constant.h), a cluster‑protocol version bump (ProtocolDefines.h), two additive virtual columns in VirtualColumnUtils.cpp (_row_id, _last_updated_sequence_number), Iceberg unit/integration tests, and an iceberg-spark-runtime bump in the integration Dockerfile. None of that is exercised by the checks that failed.

Root cause of the big red block: dataset host unreachable

Across every failing Stateless suite and the Stress test, the server started fine and then the stateful‑dataset load timed out against a fixed external host:

Code: 1000. DB::Exception: Timeout: connect timed out: 65.108.242.32:6000. (POCO_EXCEPTION)
  (in query: INSERT INTO test.hits_s3 SELECT * FROM test.hits ...)
Load job 'load table datasets.visits_v1' failed: Timeout: connect timed out: 65.108.242.32:6000 (ASYNC_LOAD_WAIT_FAILED)

The runners could not reach the dataset/S3 host 65.108.242.32:6000, so any suite that loads hits/visits/hits_s3 failed at setup. fatal.log is empty and there are no sanitizer / LOGICAL_ERROR / segfault signatures anywhere — this is environment, not code. (praktika mislabels these as a failing "Start ClickHouse Server" step / "Failures: 1/3", and the GitHub check only shows Process completed with exit code 1; the real cause is in clickhouse-server.err.log.)

Per-check breakdown

Check	What actually happened	Classification
Stateless — cas s3 / cas / db disk distributed plan / s3 storage (asan, debug, binary)	1 dataset test per suite timed out on connect timed out: 65.108.242.32:6000	Infra (network)
Stress test (amd_asan_ubsan)	datasets.visits_v1 load timeout → no test_results.tsv → exit 210; fatal.log empty	Infra (network)
Stateless (amd_debug, sequential)	Only named failure 04051_web_disk_stale_connections; the suite's own diagnosis re-ran it 10×, all passed ("failure is not reproducible")	Flaky
Integration (amd_asan_ubsan, db disk, old analyzer, 6/8)	xdist.dsession.Interrupted: session-timeout: 10800.0 sec exceeded at 92%, tests passing (no Iceberg test involved)	Infra (3h timeout)
Regression: cas_selects, cas_lightweight_delete_4, settings, swarms	Small failure counts (e.g. settings 4/1713, swarms 1/1535) in Altinity CAS/settings/swarms suites — code paths this PR does not modify, run during the same degraded window	Pre‑existing / flaky (not PR‑related)
PR (aggregate)	Rolls up the above	n/a

Recommendation

• No code change is warranted — nothing here points at the backport. Please just re‑run the failed jobs; they should clear once the 65.108.242.32:6000 dataset host is reachable and the integration shard isn't starved for time.
• One caveat worth flagging: because the integration shards were mostly skipped/timed‑out, this PR's own new Iceberg tests (test_storage_iceberg_with_spark/test_row_lineage*.py, stateless 04203_system_iceberg_files) did not get a clean validation run. A re‑run is needed to actually confirm the backport's new functionality passes — the current red is not evidence for or against it.

_{Evidence: result_pr.json, clickhouse-server.err.log and job.log for the stateless/stress suites at 868dfb9. I can't build or run ClickHouse in this environment, so this triage is from the CI reports only.}

[ianton-ru]

Can't be merged because of incompatible protocol, see discussion in private slack
https://altinity.slack.com/archives/C089554C9KP/p1789112837538069