Skip to content

fix(k8s): honor scope-configurations image pull secrets over the values.yaml default - #254

Open
pmunoz-null wants to merge 1 commit into
betafrom
fix/k8s-pull-secrets-provider-precedence
Open

fix(k8s): honor scope-configurations image pull secrets over the values.yaml default#254
pmunoz-null wants to merge 1 commit into
betafrom
fix/k8s-pull-secrets-provider-precedence

Conversation

@pmunoz-null

@pmunoz-null pmunoz-null commented Sep 10, 2026

Copy link
Copy Markdown

Summary

  • k8s/deployment/build_context resolved image pull secrets from values.yaml before the scope-configurations provider. Because k8s/values.yaml always ships IMAGE_PULL_SECRETS (ENABLED: true, SECRETS: [ecr-secret]), the provider branch was unreachable: the security.image_pull_secrets_enabled / security.image_pull_secrets properties documented in k8s/README.md had no effect.
  • Any non-EKS cluster (e.g. AKS pulling from ACR) therefore rendered pods with imagePullSecrets: [ecr-secret], a secret that does not exist there → FailedToRetrieveImagePullSecret → anonymous pull → 401ImagePullBackOff, until node-level registry credentials eventually let the pull through. Deployments succeeded late (tens of seconds to minutes) and with a misleading failure block in the logs.
  • The block is extracted into resolve_image_pull_secrets() with this priority:
    1. PULL_SECRETS env var (unchanged)
    2. provider, only when it explicitly sets image_pull_secrets_enabled
    3. values.yaml IMAGE_PULL_SECRETS — unchanged behaviour for every install that does not set the provider property
    4. disabled

Backwards compatible: nothing changes for clusters that don't set the provider property; EKS installs relying on the shipped ecr-secret default keep working. Deliberately does not touch k8s/values.yaml — removing the default there would strip imagePullSecrets from those installs.

Why get_config_value isn't used for the flag

It evaluates <path> // empty, and jq's // treats false as empty, so an explicit image_pull_secrets_enabled: false from the provider would read as "not set". The flag is read with an explicit null-check instead; the secrets list still goes through get_config_value.

Tests

The previous test (image pull secrets: PULL_SECRETS takes precedence…) asserted on an inline copy of the if/else, so it could not catch this. It is replaced by five tests that exercise the real function (extracted via the same sed pattern used for validate_status):

  • PULL_SECRETS env wins over provider and default
  • provider wins over the values.yaml default (red before this change)
  • provider can explicitly disable pull secrets (red before this change)
  • falls back to the values.yaml default when the provider is silent
  • disabled when neither provider nor default is set

bats k8s/deployment/tests/build_context.bats → 82 ok, 0 failed. Full k8s/deployment/tests → 316 ok; the 5 failures in build_deployment.bats / grpc_port_shape.bats are pre-existing on beta (they need gomplate, exit 127 in my environment) and unrelated.

Changelog

One-liner added under ## [Unreleased] (v1.16.3 is already tagged).

Not in this PR

  • Making the shipped default flavor-aware (K8S_FLAVOR is eks by default; aks/aro/gke should not inherit an ECR secret name). That changes behaviour for non-EKS installs and deserves its own PR/discussion.

@pmunoz-null
pmunoz-null force-pushed the fix/k8s-pull-secrets-provider-precedence branch from f37f3aa to 58e8d6f Compare September 10, 2026 18:41
@pmunoz-null pmunoz-null reopened this Sep 10, 2026
…es.yaml default

The image pull secret resolution in deployment/build_context checked the
values.yaml IMAGE_PULL_SECRETS before the scope-configurations provider.
Since values.yaml always ships that key (ENABLED: true, SECRETS: [ecr-secret]),
the provider branch was unreachable and the documented
security.image_pull_secrets_enabled / security.image_pull_secrets properties
had no effect. Non-EKS clusters (e.g. AKS pulling from ACR) therefore always
rendered a pod referencing a non-existent `ecr-secret`, causing
FailedToRetrieveImagePullSecret and ImagePullBackOff until node-level
credentials kicked in.

Extract the block into resolve_image_pull_secrets() and change the priority to:
  1. PULL_SECRETS env var
  2. provider, when it explicitly sets image_pull_secrets_enabled
  3. values.yaml IMAGE_PULL_SECRETS (unchanged behaviour for anyone not
     setting the provider property)
  4. disabled

The enabled flag is read with an explicit null-check instead of
get_config_value, because that helper evaluates `<path> // empty` and
turns an explicit `false` into "not set".

Tests: replace the previous test, which asserted on an inline copy of the
logic, with tests that exercise the real function via the same sed
extraction used for validate_status.
@pmunoz-null
pmunoz-null force-pushed the fix/k8s-pull-secrets-provider-precedence branch from 58e8d6f to 28737e4 Compare September 10, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant