fix(csi): give the csi-node liveness probe to the container that serves it - #215
Merged
Merged
Conversation
…es it
livenessProbePort is served by node-driver-registrar's --http-endpoint, but the
httpGet probe on it was attached to the driver container, so the driver's life
depended on a sibling's. The registrar exits by design when registration fails
("Registration process failed with error, restarting registration container"),
and the two then deadlock: the registrar exits, its port goes with it, the probe
kills the driver, the CSI socket disappears, and the restarted registrar can no
longer reach it ("Still connecting" -> "context deadline exceeded").
Seen on csi-hpe e2e clusters, where one or two of six nodes came up with the
node plugin wedged - 13 to 17 restarts, exit code 0 "Completed" for the driver
(SIGTERM from the probe) next to exit code 1 for the registrar - and no
csi.hpe.com in their CSINode while their sibling nodes were fine. Every storage
module sets livenessProbePort for its csi-node, so all of them carry this.
The probe and the startupProbe beside it now sit on the registrar, where
/healthz reports registration and a restart is the right remedy. The driver
container is left without an httpGet probe rather than given a wrong one: that
probe never reported on the driver anyway, since it stayed green once
registration had succeeded, whatever the driver was doing. A real liveness check
for the driver needs the livenessprobe sidecar, which is a separate change.
An exec probe supplied via additionalNodeLivenessProbesCmd still wins, because a
container takes only one livenessProbe. Both rules are pinned by unit tests; the
placement assertions fail against the previous template.
Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
NikolayDemchuk
self-requested a review
September 7, 2026 10:49
NikolayDemchuk
approved these changes
Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
livenessProbePortis served bynode-driver-registrar's--http-endpoint, but thehttpGetprobe on that port was attached to the driver container. The probe (and thestartupProbebeside it) now sit on the registrar, the container that actually listens.The driver container is left without an
httpGetprobe rather than given a wrong one. Anexecprobe supplied throughadditionalNodeLivenessProbesCmdstill wins, since a container takes only onelivenessProbe.This touches every module that renders a csi-node — all of them set
livenessProbePort.Why do we need it, and what problem does it solve?
The driver's life depended on a sibling container's, and the two deadlock.
node-driver-registrarexits by design when registration fails:From there: the registrar exits → its port goes with it → the probe kills the driver (
Killing Container node failed liveness probe, will be restarted) → the CSI socket disappears → the restarted registrar can no longer reach it (Still connecting→Error connecting to CSI driver: context deadline exceeded) → repeat.Observed on csi-hpe e2e clusters: on each freshly bootstrapped cluster one or two of six nodes came up with the node plugin wedged, while the sibling nodes were fine. The container statuses show both halves of the loop:
Those nodes never register the driver, so volumes fail to attach with
CSINode <node> does not contain driver csi.hpe.com.The probe never reported on the driver in the first place: once registration had succeeded it stayed green whatever the driver was doing. So nothing is lost by removing it, and a real liveness check for the driver needs the
livenessprobesidecar — a separate change.What is the expected result?
In a rendered csi-node DaemonSet,
node-driver-registrarcarries thelivenessProbe(and thestartupProbe, when the module setsstartupProbeFailureThreshold) onlivenessProbePort, and thenodecontainer carries neither:A node whose plugin cannot register must now show only
node-driver-registrarrestarting; the driver container must stay up and keep its socket, so the registrar recovers on its own once the underlying cause clears instead of the pair locking each other out.Checklist