Document self-hosted Kubernetes OOM vs scheduling failures

[warp-agent-staging]

Description

Self-hosted Kubernetes troubleshooting does not distinguish a running task pod killed for memory from a new or replacement pod that cannot schedule. Operators only get a one-line unschedulable note and a generic “ensure sufficient resources” step. Task CPU and memory are operator-owned (pod_template and optional runner instance shape), not the worker Deployment’s 100m/128Mi defaults. Exit 143 is SIGTERM and must not be documented as OOM.

Reproduction

No product reproduction was required. Current pages omit the two failure modes; worker and server code already classify them separately (OOMKilled → container_oom; PodScheduled=False/Unschedulable after unschedulable_timeout → unschedulable).

Acceptance criteria

• Troubleshooting documents two scenarios: (1) a running task container terminated with Kubernetes reason OOMKilled, and (2) a new/replacement pod that stays unschedulable (FailedScheduling / insufficient CPU or memory), with different kubectl checks and remedies.
• Resource guidance distinguishes the long-lived worker Deployment from per-task Job pods, and states that an explicit runner instance shape sets the task container’s matching CPU/memory requests and limits (overriding pod_template on those axes).
• Docs never claim 143 = OOM; they tell operators to correlate SIGTERM with pod reason, events, and eviction/drain/deadline.
• No blanket worker size. Recommend sizing to the workload (runner or pod_template) and raising only the constrained resource.
• Adjacent pages (managed Kubernetes, reference, runners, monitoring, security) do not contradict resource ownership or failed-Job cleanup.

Proposed solution direction

Add the two scenarios on the existing troubleshooting page and tighten Kubernetes resource examples on the managed Kubernetes and reference pages. Cross-link runners for per-workload shapes. Do not invent a baseline or document retry internals.

References

• Current docs: docs/src/content/docs/platform/self-hosting/troubleshooting.mdx (task failures; unschedulable one-liner), managed-kubernetes.mdx (worker.resources vs example pod_template resources), reference.mdx (unschedulable_timeout, pod_template), runners.mdx (instance shape; no self-hosted application), monitoring.mdx (oz_worker_tasks_max_concurrent), security-and-networking.mdx (cleanup wording).
• Worker (warpdotdev/oz-agent-worker main): internal/worker/kubernetes.go — applyInstanceShapeToContainer (requests=limits per axis), classifyTerminatedReason("OOMKilled") → container_oom, shouldFailUnschedulablePod (default 30s), SIGTERM guidance for exit 143, failed Jobs left for TTL (default 24h), Helm worker.resources 100m/128Mi is the daemon only.
• Server: warp-server/logic/ai/ambient_agents/workers/selfhosted/websocket.go (InstanceShape optional; container_oom and unschedulable are operator faults); runner_instance_shape.go (self-hosted shapes are not plan-capped).
• Document: worker vs task resources; OOM vs scheduling; SIGTERM ≠ OOM; workload-specific sizing via runner or pod_template; kubectl describe pod / events; unschedulable_timeout; concurrency vs node allocatable.
• Omit: customer names, retry counts/Temporal internals, invented baselines, 143 = OOM, future resume/checkpoint behavior.
• Engineering questions (do not block docs for the two scenarios): confirm public wording that failed Jobs remain until Job TTL; whether self-hosted runner shape should be documented on runners.mdx as well as Kubernetes pages.