🐛 (ZENKO-5303) make voting and priority right

.github/copilot-instructions.md

@@ -10,6 +10,7 @@ Key paths:
 - `solution/zenkoversion.yaml` — `ZenkoVersion` CR template (dashboards, policies, feature flags, capabilities, location types)
 - `solution/kafka/Dockerfile`, `solution/kafka-connect/Dockerfile` — Scality Kafka + Kafka Connect images
 - `solution-base/mongodb/` — MongoDB Helm chart, vendored from `bitnami/charts` via `git subtree`
+- `solution-base/images/` — Scality-built MongoDB images (`mongodb-sharded`, `mongodb-exporter`, `os-shell`), vendored from `bitnami/containers` via `git subtree`
 - `tests/functional/ctst/` — TypeScript Cucumber end-to-end tests
 - `tests/workflows/` — TypeScript Jest tests for CI tooling
 - `tests/zenko_tests/` — Python + Node.js integration tests
@@ -27,6 +28,7 @@ When reviewing a PR, analyze changes against the following. Post inline comments
 | **Dockerfiles** (`solution/kafka/`, `solution/kafka-connect/`...) | Base images pinned by tag or digest, no secrets baked in, no unnecessary `COPY . .`, reasonable layer count, user is non-root where possible. |
 | **Helm charts & K8s manifests** (`solution-base/mongodb/charts/`, `monitoring/`) | Resource requests/limits set, label selectors match, no hard-coded namespaces, `securityContext` present, ServiceAccount scoping minimal. Any breaking chart-value renames documented in an upgrade note. |
 | **Chart upgrade path** (`solution-base/mongodb/charts/`, `Makefile`, `how_to_upgrade.md`) | Changes under the vendored chart prefix arrive as a `git subtree` merge (squashed) plus discrete local-modification commits, not as ad-hoc edits to the tree. |
+| **Image vendoring path** (`solution-base/images/`) | Changes under a vendored image prefix arrive as a `make -C solution-base/images vendor-sync` subtree merge plus discrete local-modification commits, not as ad-hoc edits. Every `BITNAMI_<image>_REF` in `solution-base/images/Makefile` must pin a full upstream commit SHA — never a branch, tag, or `main`. |
 | **TypeScript tests** (`tests/functional/ctst/`, `tests/workflows/`) | Proper `async`/`await`, no swallowed promise rejections, Cucumber step definitions register correctly, no accidental `.only` / `.skip`, correct use of World context in ctst. |
 | **Python tests** (`tests/zenko_tests/`) | No bare `except:`, specific exception types, consistency with existing style, `requirements.txt` kept in sync. |
 | **CI workflows** (`.github/workflows/`) | Actions pinned (tag or SHA), secrets not echoed to logs, `permissions:` block scoped minimally, reusable-workflow inputs/secrets wired correctly, runner labels valid for Scality infra. |

CLAUDE.md

@@ -8,6 +8,7 @@
 - `solution/zenkoversion.yaml` — `ZenkoVersion` CR template (dashboards, policies, feature flags, capabilities, location types)
 - `solution/kafka/Dockerfile`, `solution/kafka-connect/Dockerfile` — Scality-built Kafka + Kafka Connect images
 - `solution-base/mongodb/` — MongoDB Helm charts and patches
+- `solution-base/images/` — Scality-built MongoDB images (`mongodb-sharded`, `mongodb-exporter`, `os-shell`) vendored from `bitnami/containers` via `git subtree`; upstream refs are SHA-pinned in its `Makefile`
 - `tests/functional/ctst/` — TypeScript Cucumber end-to-end tests
 - `tests/workflows/` — TypeScript Jest tests for CI tooling
 - `tests/zenko_tests/` — Python + Node.js integration tests

README.md

@@ -52,6 +52,4 @@ all configured to talk to each other.
 - Includes high availability (HA)
 - Asks for pre-existing volumes
 
-[Zenko Kubernetes Helm Chart deployment](./kubernetes)
-
 [Deploying a HA Kubernetes cluster](https://github.com/scality/metal-k8s)

monitoring/README.md

@@ -25,7 +25,7 @@
    * set the `version` of the model to `1`
    * remove `iteration` entirely from the model
    * set `annotations` to `{"list": []}`
-3. Comnit in `monitoring/*/dashboard.json`
+3. Commit in `monitoring/*/dashboard.json`
 
 
 # Modifying existing dashboards

solution-base/images/Makefile

@@ -1,21 +1,21 @@
 BITNAMI_REMOTE := bitnami-containers
 BITNAMI_REPO := https://github.com/bitnami/containers.git
-BITNAMI_UPSTREAM_MAIN_REF := $(BITNAMI_REMOTE)/main
 
 IMAGES := mongodb-sharded mongodb-exporter os-shell
 
 BITNAMI_mongodb_sharded_PATH := 8.0/debian-12
-# Latest commit on upstream main known to still contain
-# bitnami/mongodb-sharded/8.0/debian-12.
-BITNAMI_mongodb_sharded_REF := 48a109547d39cd8cf8a5d4058d832ecb5844829e
+BITNAMI_mongodb_sharded_REF := 657585595c550d4dc107a4e6cd3a598a9d284eec
 
 BITNAMI_mongodb_exporter_PATH := 0/debian-12
-BITNAMI_mongodb_exporter_REF := $(BITNAMI_UPSTREAM_MAIN_REF)
+BITNAMI_mongodb_exporter_REF := 3af1e651db624f9938f407de85387bc7fe721d49
 
 BITNAMI_os_shell_PATH := 12/debian-12
-BITNAMI_os_shell_REF := $(BITNAMI_UPSTREAM_MAIN_REF)
+BITNAMI_os_shell_REF := f6aa42546a035ccefc7a2823674e622cc9ccaaba
 
-.PHONY: create-remote fetch-remote vendor-sync $(addprefix vendor-sync-,$(IMAGES)) $(addprefix update-vendor-branch-,$(IMAGES))
+.PHONY: create-remote vendor-sync
+
+# Lock the .git/config to avoid corruption
+.NOTPARALLEL:
 
 normalize = $(subst -,_,$1)
 bitnami_path = $(BITNAMI_$(call normalize,$1)_PATH)
@@ -25,15 +25,30 @@ vendor_branch = vendor/$1/$(call bitnami_path,$1)
 create-remote:
 	@git remote get-url $(BITNAMI_REMOTE) >/dev/null 2>&1 || git remote add $(BITNAMI_REMOTE) $(BITNAMI_REPO)
 
-fetch-remote: create-remote
-	# Fetch full history from Bitnami main so subtree split can see full subtree history.
-	git fetch $(BITNAMI_REMOTE) main
-
-update-vendor-branch-%: fetch-remote
+update-vendor-branch-%: create-remote
+	# Shallow and blobless: the split only reads trees, and a full-history walk over
+	# the containers monorepo costs hours because `git subtree split` cannot restrict
+	# its rev-list to the prefix. Unrelated split roots are fine for a --squash merge.
+	git fetch --depth 1 --filter=blob:none $(BITNAMI_REMOTE) $(call bitnami_ref,$*)
 	-git branch -D $(call vendor_branch,$*)
-	git subtree split --prefix=bitnami/$*/$(call bitnami_path,$*) $(call bitnami_ref,$*) -b $(call vendor_branch,$*)
-
-vendor-sync-%: update-vendor-branch-%
+	sha=$(call bitnami_ref,$*) && \
+	wt=$$(mktemp -d) && \
+	trap 'git worktree remove --force "$$wt" 2>/dev/null' EXIT && \
+	git worktree add --no-checkout --detach "$$wt" "$$sha" && \
+	mkdir -p "$$wt"/bitnami/$*/$(call bitnami_path,$*) && \
+	git -C "$$wt" subtree split --prefix=bitnami/$*/$(call bitnami_path,$*) "$$sha" -b $(call vendor_branch,$*)
+	# `git subtree merge --squash` resolves the recorded `git-subtree-split` trailer as
+	# an object, so the split commit has to stay reachable on origin or `vendor-sync`
+	# only ever works on the clone that produced it. Keyed by upstream sha so earlier
+	# baselines are never overwritten.
+	git push origin $(call vendor_branch,$*):refs/heads/vendor-baseline/$*/$(call bitnami_ref,$*)
+
+fetch-vendor-baseline-%:
+	-git fetch --no-tags origin \
+		'refs/heads/vendor-baseline/$*/*:refs/vendor-baseline/$*/*'
+
+vendor-sync-%: fetch-vendor-baseline-% update-vendor-branch-%
+	cd "$$(git rev-parse --show-toplevel)" && \
 	git subtree merge --prefix=solution-base/images/$*/debian-12 $(call vendor_branch,$*) --squash
 
 vendor-sync: $(addprefix vendor-sync-,$(IMAGES))

solution-base/images/README.md

@@ -41,23 +41,30 @@ Do not copy files with `cp -r`. It keeps deleted files and loses merge history.
 Use Git merge semantics through `git subtree`.
 
 1. Ensure `git subtree` is available (may require installing git contrib tools).
-2. Run a single sync command:
+2. Bump `BITNAMI_<image>_REF` in `solution-base/images/Makefile` to the upstream
+   commit you want, and commit that change.
+3. From a clean working tree, run a single sync command:
 
 ```bash
 make -C solution-base/images vendor-sync
 ```
 
-This command:
+This command, per image:
 - creates the remote if needed,
-- fetches upstream refs once,
-- rebuilds all vendor branches at the same upstream point,
-- merges upstream updates for all three images.
+- fetches the pinned upstream commit,
+- rebuilds the vendor branch and publishes it to `origin` as
+  `vendor-baseline/<image>/<upstream-sha>`, so later syncs work from any clone,
+- merges upstream updates.
 
 Notes:
-- `mongodb-sharded` is pinned to a specific upstream commit in
-  `solution-base/images/Makefile`, because upstream `main` no longer contains
-  `bitnami/mongodb-sharded/8.0/debian-12`.
-- `mongodb-exporter` and `os-shell` are split from upstream `main`.
+- Every image is pinned to an explicit upstream commit; upstream `main` is never
+  used. To pick a new one, list the release commits for the image's prefix with
+  `gh api "repos/bitnami/containers/commits?path=bitnami/<image>/<path>"` and
+  check the prefix still exists at that commit.
+- `mongodb-sharded` cannot be bumped: upstream deleted
+  `bitnami/mongodb-sharded/8.0/debian-12`, so it stays pinned to the last commit
+  that still contains it.
+- You need push access to `origin`, and a clean working tree.
 
 After each upstream merge, make explicit local commits for Zenko-specific
 tweaks (for example base image pin updates, script adjustments, build changes).
@@ -76,15 +83,16 @@ Merge upstream commit YYYY
 
 To bump MongoDB:
 1. Update `solution-base/deps.yaml` `mongodb-sharded.tag`.
-2. CI passes `MONGODB_VERSION` from `deps.yaml` during `build-mongodb-images`.
+2. CI passes `MONGODB_VERSION` from `deps.yaml` during the `build-mongodb` job.
 3. For local builds, pass `--build-arg MONGODB_VERSION=<version>` explicitly.
 4. Also review the base image digest in each Dockerfile `FROM ...@sha256:...`.
    We keep the digest in Dockerfiles so dependency tooling can detect and
    propose updates.
 
 ## CI Tagging Policy
 
-- CI builds happen in `.github/workflows/end2end.yaml` (`build-mongodb-images`).
+- CI builds happen in `.github/workflows/build-mongodb.yaml`, called by
+  `end2end.yaml` and `cache-warmer.yaml`.
 - Published and consumed tags are immutable: `${VERSION}-${TREE_HASH}`.
 - Floating tags are not used by ISO build nor by tests.
 

solution-base/images/mongodb-exporter/debian-12/Dockerfile

@@ -23,7 +23,7 @@ ENV HOME="/" \
 COPY prebuildfs /
 SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-c"]
 # Install required system packages and dependencies
-RUN /bin/bash /usr/sbin/install_packages ca-certificates curl procps
+RUN install_packages ca-certificates curl procps
 RUN --mount=type=secret,id=downloads_url,env=SECRET_DOWNLOADS_URL \
     DOWNLOADS_URL=${SECRET_DOWNLOADS_URL:-${DOWNLOADS_URL}} ; \
     mkdir -p /tmp/bitnami/pkg/cache/ ; cd /tmp/bitnami/pkg/cache/ || exit 1 ; \
@@ -44,7 +44,7 @@ RUN apt-get update && apt-get upgrade -y && \
 RUN chmod g+rwX /opt/bitnami
 RUN find / -perm /6000 -type f -exec chmod a-s {} \; || true
 RUN ln -sf /opt/bitnami/mongodb-exporter/bin/mongodb_exporter /bin/mongodb_exporter
-RUN /bin/bash /usr/sbin/uninstall_packages curl
+RUN uninstall_packages curl
 
 ENV APP_VERSION="0.49.0" \
     BITNAMI_APP_NAME="mongodb-exporter" \

solution-base/images/mongodb-sharded/debian-12/Dockerfile

@@ -27,7 +27,7 @@ ENV HOME="/" \
 COPY prebuildfs /
 SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-c"]
 # Install required system packages and dependencies
-RUN /bin/bash /usr/sbin/install_packages ca-certificates curl libbrotli1 libcom-err2 libcurl4 libffi8 libgcc-s1 libgmp10 libgnutls30 libgssapi-krb5-2 libhogweed6 libidn2-0 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.5-0 libnettle8 libnghttp2-14 libp11-kit0 libpsl5 librtmp1 libsasl2-2 libssh2-1 libssl3 libtasn1-6 libunistring2 libzstd1 numactl procps zlib1g
+RUN install_packages ca-certificates curl libbrotli1 libcom-err2 libcurl4 libffi8 libgcc-s1 libgmp10 libgnutls30 libgssapi-krb5-2 libhogweed6 libidn2-0 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.5-0 libnettle8 libnghttp2-14 libp11-kit0 libpsl5 librtmp1 libsasl2-2 libssh2-1 libssl3 libtasn1-6 libunistring2 libzstd1 numactl procps zlib1g
 RUN --mount=type=secret,id=downloads_url,env=SECRET_DOWNLOADS_URL \
     DOWNLOADS_URL=${SECRET_DOWNLOADS_URL:-${DOWNLOADS_URL}} ; \
     mkdir -p /tmp/bitnami/pkg/cache/ ; cd /tmp/bitnami/pkg/cache/ || exit 1 ; \
@@ -77,8 +77,7 @@ RUN ln -s /opt/bitnami/scripts/liblog.sh /liblog.sh
 RUN ln -s /opt/bitnami/scripts/mongodb-sharded/run.sh /run.sh
 
 COPY rootfs /
-RUN find /opt/bitnami/scripts -name "*.sh" -exec chmod a+x {} +
-RUN /bin/bash /opt/bitnami/scripts/mongodb-sharded/postunpack.sh
+RUN /opt/bitnami/scripts/mongodb-sharded/postunpack.sh
 ENV APP_VERSION="${MONGODB_VERSION}" \
     BITNAMI_APP_NAME="mongodb-sharded" \
     IMAGE_REVISION="1" \

solution-base/images/mongodb-sharded/debian-12/rootfs/opt/bitnami/scripts/libmongodb.sh

@@ -910,14 +910,17 @@ mongodb_is_secondary_node_ready() {
     local -r port="${2:?port is required}"
 
     debug "Waiting for the node to be marked as secondary"
+    # mongosh prints a connection banner holding 'directConnection=true', so matching
+    # on 'true' would always succeed. Match a dedicated sentinel instead, built at
+    # runtime so the positive value never appears verbatim in the submitted script.
     result=$(
         mongodb_execute_print_output "$MONGODB_INITIAL_PRIMARY_ROOT_USER" "$MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD" "admin" "$MONGODB_INITIAL_PRIMARY_HOST" "$MONGODB_INITIAL_PRIMARY_PORT_NUMBER" <<EOF
-rs.status().members.filter(m => m.name === '$node:$port' && m.stateStr === 'SECONDARY').length === 1
+print("IS_SECONDARY_" + (rs.status().members.some(m => m.name === '$node:$port' && m.stateStr === 'SECONDARY') ? "YES" : "NO"))
 EOF
     )
     debug "$result"
 
-    grep -q "true" <<<"$result"
+    grep -q "IS_SECONDARY_YES" <<<"$result"
 }
 
 ########################
@@ -951,6 +954,35 @@ EOF
     grep -q "ok: 1" <<<"$result"
 }
 
+########################
+# Get if secondary node already has voting rights
+# Globals:
+#   MONGODB_*
+# Arguments:
+#   $1 - node
+#   $2 - port
+# Returns:
+#   Boolean
+#########################
+mongodb_secondary_node_has_voting_rights() {
+    local -r node="${1:?node is required}"
+    local -r port="${2:?port is required}"
+    local result
+
+    debug "Checking voting rights of the node"
+    # mongosh prints a connection banner holding 'directConnection=true', so matching
+    # on 'true' would always succeed. Match a dedicated sentinel instead, built at
+    # runtime so the positive value never appears verbatim in the submitted script.
+    result=$(
+        mongodb_execute_print_output "$MONGODB_INITIAL_PRIMARY_ROOT_USER" "$MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD" "admin" "$MONGODB_INITIAL_PRIMARY_HOST" "$MONGODB_INITIAL_PRIMARY_PORT_NUMBER" <<EOF
+print("HAS_VOTES_" + (rs.conf().members.some(m => m.host === '$node:$port' && m.votes > 0 && m.priority > 0) ? "YES" : "NO"))
+EOF
+    )
+    debug "$result"
+
+    grep -q "HAS_VOTES_YES" <<<"$result"
+}
+
 ########################
 # Get if hidden node is pending
 # Globals:
@@ -1187,7 +1219,15 @@ mongodb_configure_secondary() {
             exit 1
         fi
         mongodb_wait_confirmation "$node" "$port"
+    fi
 
+    # Grant voting rights to the node if it does not have them yet. This must be
+    # done even when the node is already in the cluster: a previous attempt may
+    # have added it with votes/priority 0 and failed (or been restarted) before
+    # granting voting rights, leaving the node stuck without them.
+    if mongodb_secondary_node_has_voting_rights "$node" "$port"; then
+        info "Node already has voting rights"
+    else
         # Ensure that secondary nodes do not count as voting members until they are fully initialized
         # https://docs.mongodb.com/manual/reference/method/rs.add/#behavior
         if ! retry_while "mongodb_is_secondary_node_ready $node $port" "$MONGODB_INIT_RETRY_ATTEMPTS" "$MONGODB_INIT_RETRY_DELAY"; then
@@ -1198,17 +1238,16 @@ mongodb_configure_secondary() {
         # Grant voting rights to node
         # https://docs.mongodb.com/manual/tutorial/modify-psa-replica-set-safely/
         if ! retry_while "mongodb_configure_secondary_node_voting $node $port" "$MONGODB_INIT_RETRY_ATTEMPTS" "$MONGODB_INIT_RETRY_DELAY"; then
-            error "Secondary node did not get marked as secondary"
+            error "Secondary node did not get granted voting rights"
             exit 1
         fi
+    fi
 
-        # Mark node as readable. This is necessary in cases where the PVC is lost
-        if is_boolean_yes "$MONGODB_SET_SECONDARY_OK"; then
-            mongodb_execute_print_output "$MONGODB_INITIAL_PRIMARY_ROOT_USER" "$MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD" "admin" <<EOF
+    # Mark node as readable. This is necessary in cases where the PVC is lost
+    if is_boolean_yes "$MONGODB_SET_SECONDARY_OK"; then
+        mongodb_execute_print_output "$MONGODB_INITIAL_PRIMARY_ROOT_USER" "$MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD" "admin" <<EOF
 rs.secondaryOk()
 EOF
-        fi
-
     fi
 }
 

solution-base/images/os-shell/debian-12/Dockerfile

@@ -23,7 +23,7 @@ ENV HOME="/" \
 COPY prebuildfs /
 SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-c"]
 # Install required system packages and dependencies
-RUN /bin/bash /usr/sbin/install_packages ca-certificates curl jq procps
+RUN install_packages ca-certificates curl jq procps
 RUN --mount=type=secret,id=downloads_url,env=SECRET_DOWNLOADS_URL \
     DOWNLOADS_URL=${SECRET_DOWNLOADS_URL:-${DOWNLOADS_URL}} ; \
     mkdir -p /tmp/bitnami/pkg/cache/ ; cd /tmp/bitnami/pkg/cache/ || exit 1 ; \

solution-base/mongodb/how_to_upgrade.md

@@ -28,10 +28,11 @@ See `solution-base/images/README.md` for details.
 
 ### Re-vendoring upstream Bitnami scripts
 
-When upstream Bitnami script changes need to be pulled in,
-compare `bitnami/containers` and merge upstream changes to `prebuildfs/` /
-`rootfs/` directories into the corresponding image directories. See
-`solution-base/images/README.md` for the detailed steps to upgrade the images.
+When upstream Bitnami script changes need to be pulled in, bump the
+`BITNAMI_<image>_REF` pin in `solution-base/images/Makefile` and run
+`make -C solution-base/images vendor-sync`, which merges upstream through
+`git subtree`. Do not copy files by hand. See
+`solution-base/images/README.md` for the detailed steps.
 
 ## Helm Chart Upgrade Steps
 

tests/functional/ctst/README.md

@@ -7,7 +7,7 @@ Note : Multiple Cucumber vscode extensions are available to help with developmen
 
 ## Quick Start (Github Codespaces)
 
-> **Prerequisites:** This assumes you're running in a GitHub Codespace with the devcontainer fully set up. See [.devcontainer/README.md](../../.devcontainer/README.md) for setup details.
+> **Prerequisites:** This assumes you're running in a GitHub Codespace with the devcontainer fully set up. See [.devcontainer/README.md](../../../.devcontainer/README.md) for setup details.
 
 Run tests with a tag: