Skip to content

fix: drop pause annotation from primary ScaledObject - #1972

Open
bryantvolk wants to merge 2 commits into
fluxcd:mainfrom
bryantvolk:fix/scaledobject-paused-primary
Open

bryantvolk wants to merge 2 commits into
fluxcd:mainfrom
bryantvolk:fix/scaledobject-paused-primary

Conversation

@bryantvolk

@bryantvolk bryantvolk commented Sep 15, 2026

Copy link
Copy Markdown

tldr

vibe slop, sorry. this issue is my bane

Fixes #1934

Why

When Flagger creates the primary KEDA ScaledObject, it copies annotations from the source so scaledobject.keda.sh/transfer-hpa-ownership can take over an existing HPA. If the source is already paused, that copy also includes autoscaling.keda.sh/paused-replicas: "0". Resume only clears the source, so KEDA never builds the primary HPA and the primary Deployment stays at 0 replicas while the Canary reports Succeeded.

This strips the pause annotation on primary create and leaves source pause and resume unchanged.

Scope

  • makeObjectMetaSo in pkg/canary/scaled_object_reconciler.go deletes keda.PausedReplicasAnnotation after filterMetadata.
  • Test_reconcilePrimaryScaledObject pauses the source before create and asserts the primary does not carry the pause key and still has transfer-hpa-ownership.

Out of scope: healing an already-stuck primary without recreate. Update still only rewrites metadata when the spec diffs and init is false.

Tradeoffs

Blast Radius

Canaries that use autoscalerRef.kind: ScaledObject get an unpaused primary on create and recreate. Source ScaledObjects still pause at 0 after bootstrap. After upgrade, delete each stuck -primary ScaledObject once so Flagger recreates it without the pause annotation.

Verification

Failing before the fix:

--- FAIL: Test_reconcilePrimaryScaledObject (0.00s)
    scaled_object_reconciler_test.go:35:
        	Error: map[string]string{"autoscaling.keda.sh/paused-replicas":"0", "scaledobject.keda.sh/transfer-hpa-ownership":"true"} should not contain "autoscaling.keda.sh/paused-replicas"

Passing after:

go test ./pkg/canary -run 'Test_reconcilePrimaryScaledObject|Test_pauseScaledObject|Test_resumeScaledObject' -count=1
ok  	github.com/fluxcd/flagger/pkg/canary	0.990s

go test ./... passed. make test failed in hack/verify-codegen.sh because kube_codegen.sh is missing from the local k8s.io/code-generator module. That script is unchanged by this PR.

bryantvolk and others added 2 commits September 15, 2026 16:13
Reproduce the metadata leak that leaves a newly created primary KEDA
ScaledObject paused.

Signed-off-by: Bryant Volk <bryant.volk@nominal.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
Create copies source ScaledObject annotations so
transfer-hpa-ownership can migrate an existing HPA.
That also copied autoscaling.keda.sh/paused-replicas
when the source was paused, so KEDA never built the
primary HPA.

Strip the pause annotation on create. Leave source
pause and resume unchanged.

Fixes: fluxcd#1934
Signed-off-by: Bryant Volk <bryant.volk@nominal.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Primary ScaledObject stuck paused after init when autoscalerRef is a ScaledObject

1 participant