fix(chk): let healthy keeper reconciles complete - #2059
Closed
KyriosGN0 wants to merge 1 commit into
Closed
Conversation
Signed-off-by: AvivGuiser <avivguiser@gmail.com>
KyriosGN0
marked this pull request as ready for review
August 7, 2026 17:08
Author
|
hey @alex-zaitsev could you please take a look here? Thanks! |
5 tasks
sunsingerus
added a commit
that referenced
this pull request
Sep 7, 2026
The six commits from PR #2070 were rebased onto 0.27.4 (c1c1d47..4ebf303) and reworked on top. This merge records the original PR branch as a parent so the contribution is credited; the tree is unchanged. Conflict resolution: 0.27.4's finalizeCR keeps its aborted guard, which skips the completion bookkeeping when normalize refuses a spec. The PR's persistReconcileCompleted called ReconcileComplete()/SetAncestor unconditionally and would have reverted it. Both intents are kept - the guard, and the PR's status-write error propagation (#2059). Reworked on top of the original PR: - a quorum-headroom floor: below 3 members there is no headroom to protect, so the gate only forbade the roll. A 2-node CHK could never be rolled again once upgraded - peer Ready counts are re-read during the wait; they were frozen at reconcile start, so the wait could never observe the recovery it was waiting for - a failed StatefulSet Get is surfaced rather than undercounting Ready, which flipped the pass to bootstrap and disabled both the gate and the Ready wait - membership is max(desired, ancestor), so a downscale keeps gating at the live Raft member count
Member
|
Thank you for your effort, we incorporated some of your proposals into a bigger PR. Covered in #2070 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Completedis persistedRoot cause
The CHK reconciler waited 10 seconds even when Keeper membership had not changed. More importantly, status persistence errors from both reconcile start and finalization were discarded. A failed final status write could therefore leave the resource at
InProgresswithtaskIDsStartedgrowing and notaskIDsCompleted, while the controller still reported successful reconciliation.Impact
Healthy Keeper installations can converge to and remain at
Completed. Transient status update failures are returned to controller-runtime for retry instead of being silently treated as success. No CRD or public configuration changes are required.Fixes #2035
Validation
go test -count=1 -vet=off -race ./pkg/controller/chk/... ./pkg/controller/common/statefulset/... ./pkg/apis/clickhouse-keeper.altinity.com/v1/...go test -vet=off ./pkg/controller/...python3 -m py_compile tests/e2e/test_operator.pygit diff --check upstream/0.27.3...HEADThe cluster-dependent e2e scenario was added but not run locally. Standard Go 1.26 vet remains affected by pre-existing dynamic logging format warnings.