keeper: accept node* endpoints and read <identity> for digest auth, add clickhouse.keeper_identity

[gamer22026]

fix #1546

Problem

On a cluster managed by the ClickHouse Kubernetes operator (ghcr.io/clickhouse/clickhouse-operator), clickhouse-backup 2.8.0 cannot back up RBAC or named collections. Both steps run by default (rbac_backup_always / named_collections_backup_always) and end in log.Fatal, so server --watch with watch_is_main_process: true restarts on every scheduled run and never uploads a backup. We ran five days with zero backups before the container restart count gave it away.

There are two mismatches between what keeper.Connect reads from preprocessed_configs/config.xml and what ClickHouse itself reads from the same file:

1. Endpoint element name. parseKeeperNodes selects only children named exactly node. ClickHouse accepts every <zookeeper> child whose name starts with node (src/Common/ZooKeeper/ZooKeeperArgs.cpp: key.starts_with("node")). The operator writes zookeeper: nodes: [...] in YAML, which ClickHouse renders as repeated <nodes> elements:

   FTL pkg/backup/create.go:207 > error during do RBAC backup: b.createBackupRBACReplicated error:
   keeper.Connect: /zookeeper/node not exists in /var/lib/clickhouse/preprocessed_configs/config.xml
   

2. Digest auth element. Once the connection works, credentials are only sent when found in <zookeeper><digest>, an element ClickHouse does not define. The server reads <zookeeper><identity> (key == "identity" → auth_scheme = "digest"). With Keeper ACLs in place every read then fails:

   FTL ... b.createBackupRBACReplicated error: keeper.ChildCount: ChildCount conn.Children: zk: not authenticated
   

   The operator, like many hand-written configs, also hides that value from the preprocessed file (hide_in_preprocessed / from_env), so there is nothing to read there at all.

Changes

• parseKeeperNodes accepts every element child of <zookeeper> whose name starts with node, mirroring the server rule. Per-node error messages name the element actually found (/zookeeper/nodes[0]/host not exists ...).
• keeper.Connect takes digest credentials from, in order: the new clickhouse.keeper_identity (env CLICKHOUSE_KEEPER_IDENTITY), <zookeeper><identity>, then the legacy <zookeeper><digest> (kept so existing configs keep working). The value is passed untrimmed, exactly as the server would send it, and is never logged; only its source is.
• ReadMe.md documents the option; ChangeLog.md entry under v2.8.1 BUG FIXES.

No behaviour change for configs that already worked: <node> still parses, <digest> still authenticates, the new option defaults to empty.

Tests

• TestParseKeeperNodesAcceptsNodePrefixedElements (<nodes>, <node1>, <node> mixed with session_timeout_ms / root / identity siblings), TestParseKeeperNodesWithoutEndpoints, TestParseKeeperNodesReportsElementNameOnBadHost, TestKeeperIdentityPrecedence.
• go test -short ./pkg/... passes.

Verified against a live cluster

ClickHouse 26.7.3.19, clickhouse-operator v0.0.7, 3-node Keeper with digest ACLs, user_directories.replicated and named_collections_storage: keeper_encrypted. This branch cross-compiled and run inside the sidecar next to the shipped 2.8.0:

2.8.0    create --rbac-only                            EXIT=1  keeper.Connect: /zookeeper/node not exists in .../config.xml
branch   create --rbac-only, no identity               EXIT=1  keeperHosts=[3 hosts] ... zk: not authenticated
branch   create --rbac-only, CLICKHOUSE_KEEPER_IDENTITY set   EXIT=0
         INF keeper digest auth from clickhouse.keeper_identity
         WRN /clickhouse/access/uuid have no children, skip Dump
         INF done createBackupRBAC, size=0B
         INF keeper.Dump /clickhouse/named_collections -> .../named_collections/named_collections.jsonl
         INF done createBackupNamedCollections, size=23B

Possible follow-up, not in this PR

createConfigsNamedCollectionsAndRBACIfNecessary uses log.Fatal for these auxiliary steps, which turns a config-parsing problem into a total loss of data backups under watch. Returning the error instead would let the data backup proceed and surface the failure through the normal status and metrics paths. That could be a separate change if maintainers agree.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KqygQvwaxVCuS1EjCJUqxG

[gamer22026]

CI note: the Testflows matrix on the first run failed in one place only, the cli suite's default config snapshot, because default-config now prints keeper_identity: "". Every other suite passed (the other engines failures are inside its expected-failure materializedmysql scenarios, same as on master). c2d8209 updates the snapshot; regenerated from the branch binary with the test's own filter, the only diff against the stored value is that one line.

The re-run for c2d8209 is waiting for workflow approval since this comes from a fork.

Update: the first run has finished. All 16 Go integration Test jobs (ClickHouse 1.1.54394 through 26.8) and Testflows FIPS passed on b1ad467; only the Testflows cli snapshot failed, fixed in c2d8209.

[gamer22026]

Re-run for c2d8209 (https://github.com/Altinity/clickhouse-backup/actions/runs/34381311882): Testflows 11/11 green, so the snapshot fix is confirmed; Go integration 15/16 green.

The one failure is Test (1.27, 26.8) → TestCustomKopia, failing in kopia snapshot verify with object … is backed by missing blob p…-s… (four objects, same blob) after 1253 s. That is the kopia/MinIO missing-blob flake that #1479 added diagnostics for; it does not touch pkg/keeper or any code in this PR, and the identical job passed on the first run (b1ad467, same Go code — only the Testflows snapshot file changed between the runs). I cannot re-run a single job from a fork, so a maintainer re-run of the failed job should clear it.