Orchestrator restart terminally fails in-flight items instead of requeueing them; pause does not drain - #483
Merged
Merged
Conversation
…add queue drain A work item whose worker died without a preempt checkpoint is an infrastructure loss, not an item failure. All recovery surfaces (periodic dead-worker reaper, startup stranded sweep, legacy startup replay, graceful-shutdown fallback, failed sandbox resume) now return the item to Queued preserving its work branch without consuming RecoveryAttempts and never transition to Failed or AbandonedAfterRecoveryAttempts. Checkpointed turns still resume with bounded accounting; genuine terminal failures are left untouched. Adds POST /queue/drain (pause-and-wait with explicit timeoutSeconds) so an operator can restart without interrupting running work, and documents that plain pause does not wait. QueueDrain.WaitForQuiescenceAsync is unit tested; service-level test covers drain-waits vs pause-returns-immediately. Docs: recovery matrix, worker-pool pause-vs-drain, safe-restart sequence, POST /queue/drain API reference. Verification: 516 affected tests pass; solution builds warnings-clean; gitleaks clean (semgrep not installed in this environment). CodeyBox-Prompt-Revision: 1 Co-Authored-By: CodeyBox <noreply@codeybox.invalid>
…ays Failed StartupResumeApiAvailabilityTests require a suspended-VM resume that throws, hangs, or times out to mark the item Failed: the resume was attempted and the suspended state itself is unrecoverable, which is a genuine resume failure, not a plain worker loss. The previous change routed that path through the new infrastructure requeue and broke those tests (item stayed Queued). Restore TryBuildWorkingWithoutPreemptFailure as the resume-service-only failure builder and route the !resumeSucceeded branch back through it, with a doc comment scoping it to resume failure. Plain checkpoint-less worker death (DeadWorkerReaper, startup replay, shutdown fallback) keeps the budget-preserving requeue. Revert the SandboxSuspendResumeTests expectation flips for resume failure/timeout/cancellation. Keep a defensive Failed persist-without-enqueue branch in startup replay and document the resume-failure exception in the recovery matrix. Verification: orchestrator builds warnings-clean; 302 recovery tests, 5 StartupResumeApiAvailabilityTests, and 48 UAT/shutdown/drain tests pass. CodeyBox-Prompt-Revision: 1 Co-Authored-By: CodeyBox <noreply@codeybox.invalid>
…OperatorInput Worker death without a preempt checkpoint requeues without consuming RecoveryAttempts, but a poison input that deterministically kills every worker would retry forever. Track ConsecutiveInfrastructureRecoveries (persisted, reset on real progress and manual retry/resume) and park at NeedsOperatorInput past MaxConsecutiveInfrastructureRecoveries (default 20, hot-reloadable via CodeyBox:DeadWorker). Also fix stale recovery doc, remove dead Failed fallback, extract queue-reason/drain constants, and rename noPreemptFailedReason to noPreemptRequeueReason. CodeyBox-Prompt-Revision: 1 Co-Authored-By: CodeyBox <noreply@codeybox.invalid>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated via CodeyBox — work item a4a9502798624be28727dd8be1740690
Initiated by CodeyBox operator
Co-Authored-By: CodeyBox noreply@codeybox.invalid
🤖 Generated with CodeyBox