docs: verify MLflow delivery and recover existing evaluation exports

[chadvoegele]

What does this PR do?

Type of change: documentation.

Chad's Agent: Separate evaluation completion from verified MLflow delivery. Add a ModelOpt-owned gate to verify per-task identity, scores, configuration, and diagnostic artifacts; recover exports from existing results using installed CLI help; avoid duplicate exports; preserve sanitized evidence when delivery is blocked.

Motivation: completed evaluations can lose delivery when their separate CPU export jobs fail to import the export image. This must not trigger another evaluation or a false success report.

Both evaluator definitions return verified URLs and invocation IDs with independent evaluation/export outcomes. Vendored skills and nel-next's explicit push workflow are unchanged.

Usage

Follow evaluation/references/mlflow-verification.md after completed-run validation. No API changes.

Testing

• Evaluator-definition parity: uv run --no-project --with pytest pytest -o addopts='' -q plugins/modelopt/skills/day0-release/tests/test_agent_definitions.py — 1 passed (existing timeout-config warning).
• Pre-commit on all five changed files — passed.
• JSON scenario validation and git diff --check — passed.
• Added four behavioral scenarios: auto-export verification, manual/partial recovery, blocked delivery before cleanup, and nel-next push verification. These are not executed agent evaluations; no live MLflow/cluster export was run.

Before your PR is "Ready for review"

• Is this change backward compatible?: ✅
• Copied code or new PIP dependency guidance: N/A
• Necessary tests?: ✅ — focused scenarios and existing parity test
• Changelog?: N/A — guidance only
• Claude approval?: N/A — draft

Additional Information

Coordinated scope with the parent-authorization agent (#2473) and truncation-policy agent. This PR does not change acceptance thresholds or submission authorization; export recovery never starts another evaluation. Keep the standalone export paragraph in both evaluator definitions when integrating their handoff edits.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.25%. Comparing base (cf1f48f) to head (71bdddf).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2474      +/-   ##
==========================================
- Coverage   70.91%   70.25%   -0.66%     
==========================================
  Files         600      601       +1     
  Lines       65987    67132    +1145     
==========================================
+ Hits        46794    47166     +372     
- Misses      19193    19966     +773     

Flag	Coverage Δ
unit	58.10% <ø> (-0.09%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[chadvoegele]

Chad's Agent — independent review

Reviewed head 0e8737e489a6e7709e27f893bbb51dd00e15bafe.

• P2 — Put the pre-upload safeguard on every submission path. evaluation/SKILL.md:158–159: this check lives in Step 2, but the shortcut explicitly skips Step 2 and existing-config runs jump to Step 8. Both can enable auto-export before reaching the reference in Step 9. NEL 0.2.6 submits the CPU exporter from the evaluation script without an intervening agent review, so a post-run scan cannot satisfy the new before-upload requirement. Move the prerequisite into the common pre-submit path; where generated artifacts cannot be checked before automatic upload, disable auto-export and use checked/sanitized manual delivery.

• P2 — Connect the nel-next branch to the new verification gate. evaluation/SKILL.md:566–569: the nel-next branch explicitly says “Steps 1–9 don't apply,” and its referenced run flow still ends at mlflow-push. Saying the gate also applies here, inside excluded Step 9, leaves standalone evaluation users with contradictory instructions. Add the gate directly after push in that branch/reference. This matters operationally: the current wrapper stages only eval-*.json, so a successful push does not establish delivery of the required logs/diagnostic evidence. The new evaluator-agent paragraph covers delegated runs, not every invocation of this skill.

Checks: entire five-file diff; surrounding skills, recipes, wrapper and instructions; legacy NEL 0.2.6 exporter/auto-export implementation; nel-next exporter at the wrapper's pinned 4d081325… commit. Agent-definition parity: 1 passed; scenario JSON parsing and diff whitespace check: passed. An offline probe with MLflow calls stubbed confirmed nel-next identity/metric handling and that bundle-only staging supplies no logs even with copy_logs=true.

Limits: no live cluster/MLflow exports or behavioral agent scenarios executed. No checkout changes or fixes made.

[chadvoegele]

Chad's Agent — review fixes

Addressed both findings in 71bdddf:

• Moved the pre-upload safeguard from Step 2 to common Step 8, covering shortcut and existing configs. The canonical reference now requires disabling auto-export before canary/full submission when generated artifacts cannot be checked before upload, retaining checked/sanitized manual delivery.
• Routed nel-next directly through checked push → canonical delivery verification. Documented JSON-only staging and repair of missing sanitized diagnostics on identified MLflow runs; unstaged logs are not supplied by copy_logs=true. No wrapper changes or duplicated verification checklist.

Checks: changed-file pre-commit passed; agent-definition parity 1 passed; static submission/gate routing, nine scenario definitions, and git diff --check passed. Added/updated existing-config, shortcut, and nel-next evidence scenarios; behavioral agent scenarios and live cluster/MLflow exports were not run.

Pushed without rebasing: remote head was unchanged and PR mergeable. No merge performed.