Skip to content

fix(recovery): bound overlapping incident retries to original plus two - #432

Merged
rldyourmnd merged 1 commit into
mainfrom
fix/recovery-incident-retry-budget-20260908
Sep 8, 2026
Merged

fix(recovery): bound overlapping incident retries to original plus two#432
rldyourmnd merged 1 commit into
mainfrom
fix/recovery-incident-retry-budget-20260908

Conversation

@rldyourmnd

Copy link
Copy Markdown
Contributor

Summary

  • Bound scheduler recovery to the original attempt plus two infrastructure retries for overlapping stuck subjects.
  • A new attempt hash, process restart, or extra max subject no longer resets that budget.
  • Cooldown and sibling-busy stay unhealthy; incomplete recovery does not become successful.
  • FileStore.Begin single-active, original-subject partition, and malformed-progress handling are unchanged.

Test plan

  • GOTOOLCHAIN=go1.26.7 go test -race -count=1 ./internal/schedulerrecovery
  • Fourth overlapping Tick must emit recovery-retry-budget-exhausted and not restart.
  • Successful overlapping recovery must reset the count.
  • This is source only. Do not treat merge as fleet adoption.

A new attempt hash after cooldown restarted the manager against the same stuck
subjects. Count unfinished overlapping recoveries in durable history, keep
cooldown and busy unhealthy, and refuse a fourth attempt without calling the
restart.

Signed-off-by: rldyourmnd <danil@nddev.it.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@rldyourmnd
rldyourmnd merged commit b10db6b into main Sep 8, 2026
10 checks passed
@rldyourmnd
rldyourmnd deleted the fix/recovery-incident-retry-budget-20260908 branch September 8, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant