fix: drop pause annotation from primary ScaledObject - #1972
Open
bryantvolk wants to merge 2 commits into
Open
bryantvolk wants to merge 2 commits into
bryantvolk wants to merge 2 commits into
Conversation
Reproduce the metadata leak that leaves a newly created primary KEDA ScaledObject paused. Signed-off-by: Bryant Volk <bryant.volk@nominal.io> Co-authored-by: Cursor <cursoragent@cursor.com>
Create copies source ScaledObject annotations so transfer-hpa-ownership can migrate an existing HPA. That also copied autoscaling.keda.sh/paused-replicas when the source was paused, so KEDA never built the primary HPA. Strip the pause annotation on create. Leave source pause and resume unchanged. Fixes: fluxcd#1934 Signed-off-by: Bryant Volk <bryant.volk@nominal.io> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tldr
vibe slop, sorry. this issue is my bane
Fixes #1934
Why
When Flagger creates the primary KEDA ScaledObject, it copies annotations from the source so
scaledobject.keda.sh/transfer-hpa-ownershipcan take over an existing HPA. If the source is already paused, that copy also includesautoscaling.keda.sh/paused-replicas: "0". Resume only clears the source, so KEDA never builds the primary HPA and the primary Deployment stays at 0 replicas while the Canary reports Succeeded.This strips the pause annotation on primary create and leaves source pause and resume unchanged.
Scope
makeObjectMetaSoinpkg/canary/scaled_object_reconciler.godeleteskeda.PausedReplicasAnnotationafterfilterMetadata.Test_reconcilePrimaryScaledObjectpauses the source before create and asserts the primary does not carry the pause key and still has transfer-hpa-ownership.Out of scope: healing an already-stuck primary without recreate. Update still only rewrites metadata when the spec diffs and
initis false.Tradeoffs
includeLabelsByPrefixon create. Default--include-label-prefixis empty, so that filter would droptransfer-hpa-ownershipand regress KEDA Scaled Object Primary creation failed due to workload already managed by the hpa #1646.Blast Radius
Canaries that use
autoscalerRef.kind: ScaledObjectget an unpaused primary on create and recreate. Source ScaledObjects still pause at 0 after bootstrap. After upgrade, delete each stuck-primaryScaledObject once so Flagger recreates it without the pause annotation.Verification
Failing before the fix:
Passing after:
go test ./...passed.make testfailed inhack/verify-codegen.shbecausekube_codegen.shis missing from the localk8s.io/code-generatormodule. That script is unchanged by this PR.