Skip to content

fix(csi): give the csi-node liveness probe to the container that serves it - #215

Merged
duckhawk merged 1 commit into
mainfrom
fix/csi-node-liveness-own-container
Sep 7, 2026
Merged

duckhawk merged 1 commit into
mainfrom
fix/csi-node-liveness-own-container

Conversation

@duckhawk

@duckhawk duckhawk commented Sep 7, 2026

Copy link
Copy Markdown
Member

Description

livenessProbePort is served by node-driver-registrar's --http-endpoint, but the httpGet probe on that port was attached to the driver container. The probe (and the startupProbe beside it) now sit on the registrar, the container that actually listens.

The driver container is left without an httpGet probe rather than given a wrong one. An exec probe supplied through additionalNodeLivenessProbesCmd still wins, since a container takes only one livenessProbe.

This touches every module that renders a csi-node — all of them set livenessProbePort.

Why do we need it, and what problem does it solve?

The driver's life depended on a sibling container's, and the two deadlock.

node-driver-registrar exits by design when registration fails:

Received NotifyRegistrationStatus call status="error:RegisterPlugin error -- plugin registration failed with err: ..."
Registration process failed with error, restarting registration container

From there: the registrar exits → its port goes with it → the probe kills the driver (Killing Container node failed liveness probe, will be restarted) → the CSI socket disappears → the restarted registrar can no longer reach it (Still connectingError connecting to CSI driver: context deadline exceeded) → repeat.

Observed on csi-hpe e2e clusters: on each freshly bootstrapped cluster one or two of six nodes came up with the node plugin wedged, while the sibling nodes were fine. The container statuses show both halves of the loop:

node                   restarts=7  lastExit=0 Completed   # SIGTERM from the probe
node-driver-registrar  restarts=7  lastExit=1 Error
Unhealthy  Liveness probe failed: Get "http://10.211.1.80:4232/healthz": connect: connection refused

Those nodes never register the driver, so volumes fail to attach with CSINode <node> does not contain driver csi.hpe.com.

The probe never reported on the driver in the first place: once registration had succeeded it stayed green whatever the driver was doing. So nothing is lost by removing it, and a real liveness check for the driver needs the livenessprobe sidecar — a separate change.

What is the expected result?

In a rendered csi-node DaemonSet, node-driver-registrar carries the livenessProbe (and the startupProbe, when the module sets startupProbeFailureThreshold) on livenessProbePort, and the node container carries neither:

- name: node-driver-registrar
  livenessProbe:
    httpGet: {path: /healthz, port: 4232}
- name: node
  # no httpGet probe

A node whose plugin cannot register must now show only node-driver-registrar restarting; the driver container must stay up and keep its socket, so the registrar recovers on its own once the underlying cause clears instead of the pair locking each other out.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

…es it

livenessProbePort is served by node-driver-registrar's --http-endpoint, but the
httpGet probe on it was attached to the driver container, so the driver's life
depended on a sibling's. The registrar exits by design when registration fails
("Registration process failed with error, restarting registration container"),
and the two then deadlock: the registrar exits, its port goes with it, the probe
kills the driver, the CSI socket disappears, and the restarted registrar can no
longer reach it ("Still connecting" -> "context deadline exceeded").

Seen on csi-hpe e2e clusters, where one or two of six nodes came up with the
node plugin wedged - 13 to 17 restarts, exit code 0 "Completed" for the driver
(SIGTERM from the probe) next to exit code 1 for the registrar - and no
csi.hpe.com in their CSINode while their sibling nodes were fine. Every storage
module sets livenessProbePort for its csi-node, so all of them carry this.

The probe and the startupProbe beside it now sit on the registrar, where
/healthz reports registration and a restart is the right remedy. The driver
container is left without an httpGet probe rather than given a wrong one: that
probe never reported on the driver anyway, since it stayed green once
registration had succeeded, whatever the driver was doing. A real liveness check
for the driver needs the livenessprobe sidecar, which is a separate change.

An exec probe supplied via additionalNodeLivenessProbesCmd still wins, because a
container takes only one livenessProbe. Both rules are pinned by unit tests; the
placement assertions fail against the previous template.

Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
@NikolayDemchuk
NikolayDemchuk self-requested a review September 7, 2026 10:49
@duckhawk
duckhawk merged commit 1eaa866 into main Sep 7, 2026
4 checks passed
@duckhawk
duckhawk deleted the fix/csi-node-liveness-own-container branch September 7, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants