Skip to content

Fail closed on unsafe incremental resumes - #57

Open
davanstrien wants to merge 5 commits into
mainfrom
fix/safe-resume-provenance
Open

davanstrien wants to merge 5 commits into
mainfrom
fix/safe-resume-provenance

Conversation

@davanstrien

@davanstrien davanstrien commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Legacy incremental runs could reuse verdicts after source data, OCR output selection, or judge settings changed. This PR stamps every comparison and checkpoint with per-pair provenance and stops unsafe resumes before reusing those verdicts.

  • Fingerprint sampled page identities/images and each OCR column's contents; include source repository/split, config-to-model mapping, original resolved output column, judge backend or jury, sample selection, prompt, preprocessing, token cap, and minimum text threshold.
  • Reuse unchanged pairs when adding a new model. Comparisons-only checkpoints can resume without a completed metadata row, and matching row provenance takes precedence over stale completed-run metadata.
  • Require --full-rejudge or a new results repository for legacy unstamped or mismatched current-grid comparisons. Preserve historical comparisons outside the current rankable grid.
  • Publish only one-way judge-spec hashes so endpoint credentials and private URLs are not exposed.

Updated against current main, including merged #55 and #56. The #55 model/output resolver now also supplies the original OCR column to provenance, so switching output columns is detected even when their text and model identity are identical. The README documents the migration.

Validation: 651 tests passed locally; Ruff, ty, and git diff --check passed. Regression coverage includes unchanged config resumes, adding a new model, incomplete appended inference metadata, changed output columns/text/judge/split, legacy results, and comparisons-only checkpoints.

@davanstrien
davanstrien changed the base branch from fix/split-end-to-end to main August 24, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant