CLEAN-96: retry transient LightRAG failures without a person

[mykyta-torbynskyi]

Jira: CLEAN-96. Seven commits, ranch first.

Problem. Bedrock answers 503 in waves of 10-15 minutes (dev, 2026-09-17 00:14-00:27: 170 in a row). LightRAG's own retry is five attempts a few seconds apart, so it gives up inside the wave and marks the document failed; Ranch records that and nothing heals it. An Index run or a Reindex re-uploads, LightRAG refuses the duplicate of the failed copy it still holds (Original doc_id: X, Status: failed), and the row fails again. Nine good documents sat red overnight; the fix was a person with a port-forward and POST /documents/reprocess_failed.

Change.

• source/domain/indexFailure.ts: a failure message is transient (the Bedrock wrapper's RetryError, a LightRAG 5xx, a lost connection, a refusal naming a failed original) or permanent (only whitespace, no content, anything unknown, on purpose). Three automatic retries after 5, 15 and 60 minutes, then an honest failed.
• Source.indexAttempts, Source.indexRetryAt (one migration, existing rows untouched). Every failed write goes through one recordFailure; success and a manual retry reset the count.
• IndexReconcileService gets a third step: due rows on an idle pipeline go back in flight under the handle LightRAG really holds (a refusal's original, not the rejected copy) and the pipeline is nudged with reprocess_failed; a document LightRAG no longer holds is uploaded again, within the attempt budget. The nudge cooldown is now per knowledge and shared with the stalled-pipeline nudge, which also became per knowledge (it used to restart the default instance only).
• A second gap found while reading: when the reconciler met a LightRAG-side failed on a row it was confirming, it dropped the handle and wrote nothing, leaving the row processing forever (the "Indexing… for four days" shape). It now records the failure and keeps the handle. LightRAG's updated_at on the document tells a fresh failure from one the row was already re-queued over.
• Fourth derived source status retrying through filter, counts, DTO and admin (badge, filter option, "Attempt 2 of 4, next try 14:32" on the row). failedCount counts terminal failures only.
• Reindex on a row whose document LightRAG still holds schedules the retry (fresh count, due now) instead of re-uploading into the refusal; the admin sources table gets a Retry button on failed and retrying rows (the store had reindexSource, nothing called it). The single-row wait no longer marks a slow document failed on timeout; it leaves it in flight for the reconciler, like the batch path.
• retryingCount travels next to failedCount and processingCount on the knowledge, the overview and the admin, so a retrying row is neither "failed" nor "never sent". The list gateway subtracts it from the stored failed states so list and detail agree.
• The confirm pass tells a fresh LightRAG verdict from the one a row was re-queued over by LightRAG's own updated_at on that verdict, stored on the row as indexRequeuedOverAt (a third column, same migration). Attempts are incremented in the database and a failure already recorded with a future slot is not charged twice, so an index run overlapping a reconcile pass cannot spend two retries on one verdict. One pass retries at most 50 rows.
• reins README: a section on the above.

Out of scope, noted in the ticket: an Index run itself nudging the pipeline (it converges through the retry path at the cost of one refused upload and five minutes; a row that already has a retry slot is reported without spending it), and LightRAG concurrency (gitops).

Verified: bun run build, generate:swagger boots (DI), bunx jest on the reins slice after the last commit: 19 suites, 212 tests green; the whole API before the review fixes: 81 suites green, with the bridle documentText.extractor.spec.ts suite red on main for the known pdf.js-in-jest reason, untouched here. Changed .vue files compile with the repo's @vue/compiler-sfc. New specs: classification (27 cases from real log lines), recording and retry paths in the gateway including the whole fail/retry/fail loop to the last attempt, the reconciler's retry step across two bases and the shared cooldown, mapper, failure summary, overview counts.

On dev after the deploy: nothing to click. The next Bedrock wave should leave retrying N source(s) in <base>: ... in the API log and rows going retrying -> processing -> indexed on their own, failedCount staying at zero.