feat(k8s): add image-readiness preflight for VERL training

[baibizhe (SII) (baibizhe)]

Summary

• publish a leased intersection of images reported by Ready, schedulable Kubernetes nodes
• filter training and validation rows by the images in their rendered Jobs before Ray initializes
• fail guarded rollouts promptly when fresh readiness data is unavailable or an image is missing
• keep the feature disabled by default and preserve the source JSONL files

Motivation

With imagePullPolicy: IfNotPresent, a missing locally built :openai image can leave a rollout in ImagePullBackOff while GPU training waits until the rollout timeout. In the setup reported in #567, preflight identified 190 of 6,343 training rows and 126 of 474 validation rows whose rendered Jobs referenced unavailable images.

Kubelet limits Node.status.images to 50 entries by default, so the documentation explains using nodeStatusMaxImages: -1 for large preloaded-image sets.

Validation

• 54 focused tests passed
• 121 functional repository tests and 3 isolated package-metadata tests passed
• Ruff check and format passed
• BasedPyright reported 0 errors and 0 warnings
• independent code review found no critical or important issues

Closes #567

[baibizhe (SII) (baibizhe)]

@microsoft-github-policy-service agree

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.