docs: verify MLflow delivery and recover existing evaluation exports

.codex/agents/modelopt_model_evaluator.toml

@@ -13,5 +13,7 @@ Before acting, load these Model Optimizer instructions:
 
 Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run.
 
+Apply `evaluation/references/mlflow-verification.md` independently of evaluation acceptance. Recover export from existing results only; never submit another evaluation to repair delivery. In `MLflow`, report each task's export outcome and verified run URL (or none verified), mapped to its invocation ID. Keep evaluation and export outcomes separate; report blockers and preserved evidence paths when delivery fails.
+
 Return only a concise handoff with these headings: `Status`, `Evaluation role`, `Checkpoint`, `Configuration`, `Results`, `Validation`, `MLflow`, `Artifacts`, and `Blockers`. Include invocation IDs, task-to-score mappings, score fields, sample accounting, and absolute paths. Do not return raw logs.
 """

plugins/modelopt/agents/modelopt-model-evaluator.md

@@ -18,4 +18,6 @@ Before acting, load these Model Optimizer instructions:
 
 Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run.
 
+Apply `evaluation/references/mlflow-verification.md` independently of evaluation acceptance. Recover export from existing results only; never submit another evaluation to repair delivery. In `MLflow`, report each task's export outcome and verified run URL (or none verified), mapped to its invocation ID. Keep evaluation and export outcomes separate; report blockers and preserved evidence paths when delivery fails.
+
 Return only a concise handoff with these headings: `Status`, `Evaluation role`, `Checkpoint`, `Configuration`, `Results`, `Validation`, `MLflow`, `Artifacts`, and `Blockers`. Include invocation IDs, task-to-score mappings, score fields, sample accounting, and absolute paths. Do not return raw logs.

plugins/modelopt/skills/evaluation/SKILL.md

@@ -43,7 +43,7 @@ for one, do **not** add it to a 0.2.6 `evaluation.tasks` list — instead:
 1. Read **`references/nel-next.md`** (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe `recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md`; start from `recipes/examples/example_eval_next.yaml`.
 2. Isolated nel-next venv: `"$SKILL_DIR/scripts/nel-next.sh" --setup-only` (keeps 0.2.6 `nel` untouched).
 3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`) into `.env`; always include the `output.export_config.mlflow` block.
-4. Dry-run → canary → full (`nel-next.sh eval run`), then **push to MLflow** — SLURM doesn't auto-export, so run `nel-next.sh mlflow-push -r <run_id> -c <cfg>` after (config-driven; see `references/nel-next.md`).
+4. Follow `references/nel-next.md`'s run flow: dry-run → canary → full → checked MLflow push → delivery verification.
 
 Steps 1–9 below are currently validated with 0.2.6 — use them for everything else.
 
@@ -155,7 +155,7 @@ Ask the 5 questions via AskUserQuestion (categories must match `nel skills build
 
 1. **Execution:** Local / SLURM
 2. **Deployment:** None (External) / vLLM / SGLang / NIM / TRT-LLM. Prefer vLLM unless the user/card says otherwise.
-3. **Auto-export:** None / MLflow / wandb
+3. **Auto-export:** None / MLflow / wandb.
 4. **Model type:** Base / Chat / Reasoning
 5. **Benchmarks** (multi-select): standard / code / math_reasoning / safety / multilingual
 
@@ -501,6 +501,10 @@ Add credentials per the common skill's `slurm-setup.md` §6 if missing. If you c
 
 Run directly when the user asked to launch; otherwise ask before submitting.
 
+**Before submitting any canary or full run** (including shortcut and existing
+configs), apply `references/mlflow-verification.md#before-upload`: disable
+auto-export if generated artifacts cannot be checked before automatic upload.
+
 **Env setup:** `.env` is normally already created and filled back in Step 1 (via `modelopttools:eval-config`), at the **workspace root** — the dir you run `nel` from, not under the skill dir. Ensure it exists and source it — do **not** clobber an existing `.env`:
 
 ```bash
@@ -558,10 +562,14 @@ Remove `limit_samples` overrides; keep canary-validated parallelism. If the cana
 
 ---
 
-### Step 9 — Verify completed run
+### Step 9 — Verify completed run and MLflow delivery
 
 Before pulling/reporting scores, validate the run. Read `references/run-validation.md` for NEL timeout/resume behavior, completed-run validation, diagnostics, and score harvesting. For a baseline that will be compared with a candidate, also perform its **External Baseline Sanity Check** before a success verdict, then hand the validated runs to `compare-results` for baseline-vs-candidate deltas.
 
+Then apply `references/mlflow-verification.md`: verify each task's actual MLflow
+run, recover failed/incomplete exports from existing results without rerunning
+evaluation, and report evaluation and export outcomes separately.
+
 ---
 
 Issues: <https://github.com/NVIDIA-NeMo/Evaluator/issues> · <https://github.com/NVIDIA-NeMo/Evaluator/discussions>

plugins/modelopt/skills/evaluation/references/mlflow-verification.md

@@ -0,0 +1,91 @@
+# Verify MLflow delivery
+
+Evaluation completion and MLflow delivery are separate outcomes. Apply this gate
+per invocation/task after run validation, before handoff or cleanup. Use
+`launching-evals` to locate existing results and `accessing-mlflow` to query the
+tracking server; those vendored skills remain unchanged.
+
+## Before upload
+
+Before any upload (automatic or manual), secret-scan all outgoing artifacts,
+including result bundles, configs and logs; redact credentials, tokens,
+authorization headers, and secret-bearing URLs. Never upload `.env`/secret files.
+Preserve non-secret methodology/provenance in sanitized copies; if safe upload
+cannot be established, report export blocked.
+
+Before submission, if generated artifacts cannot be checked before automatic
+upload, disable auto-export (`execution.auto_export.destinations: []` for NEL
+0.2.6) for both canary and full runs. Keep the MLflow export settings for checked,
+sanitized manual delivery from existing results below. A post-run scan cannot
+satisfy this prerequisite: 0.2.6 submits its exporter without an agent-review pause.
+
+## Verify before exporting again
+
+1. Record the invocation ID, task/job identity, evaluation outcome, and result
+   paths. Check the separate auto-export job's state and logs: evaluation
+   `SUCCESS` does not prove export succeeded. If export is queued/running, monitor
+   it or report export pending; do not start a competing export.
+2. Search the configured MLflow server/experiment by `invocation_id` and task
+   identity, not display name alone. Search other experiments if necessary.
+   Distinguish canary/full runs and repeated tasks. For nel-next, use its run ID
+   and bundle `job_id` identity instead. An inaccessible server is **unverified**,
+   not evidence that no run exists.
+3. Inspect each actual run's metrics and artifact contents, not just its status
+   or artifact listing. Verify the canonical score field/value against the
+   validated results, the evaluated checkpoint and resolved configuration
+   (including task overrides), and sample/scoring coverage. Check required
+   diagnostics: result file, resolved config, runtime metrics when produced,
+   and relevant client/server/SLURM/judge logs or compact diagnostic evidence
+   supporting the validation summary. Check benchmark-specific artifacts where
+   required. Record genuinely inapplicable diagnostics; missing required
+   evidence means incomplete export. Artifact paths vary: discover them.
+4. Reuse a usable run. Return its verified run URL and invocation/task mapping;
+   never infer a run URL from an invocation ID or accept a same-named run.
+
+## Recover existing results only
+
+If auto-export failed or the run is incomplete, first confirm no exporter is
+still active and recheck MLflow immediately before recovery. Inspect the
+**installed** launcher's version and help (e.g. `nel --version`, `nel --help`,
+then `nel export --help` if that subcommand exists). Use only the destination,
+invocation/task selection, config, and artifact/log options that version supports.
+
+Manually export the **existing invocation/results** to MLflow. Do not run or
+resume evaluation to repair export. Diagnose the export-specific failure: for
+example, a CPU export job's container-import failure needs an available,
+compatible export image or supported local export, not another GPU evaluation.
+Change only export settings using the installed version's supported mechanism;
+retain the original evaluation configuration and record recovery settings
+separately. A local summary export is not MLflow delivery or an artifact backup.
+
+If a partial run exists, prefer a supported update/repair of that run. Do not
+assume retries are idempotent: establish the exporter's behavior first. If a
+replacement is unavoidable, record both run IDs and identify the verified
+replacement; do not delete the original without authorization. After recovery,
+repeat the full verification above. Export exit code zero alone is insufficient.
+
+**nel-next:** Preserve the explicit `nel-next.sh mlflow-push -r <run_id> -c <cfg>`
+workflow in `nel-next.md`; SLURM does not auto-export there. The wrapper stages
+only `eval-*.json`, not logs; `copy_logs=true` cannot supply unstaged evidence.
+Verify the pushed runs, then attach missing sanitized configs/logs or compact
+diagnostic evidence to the identified runs using a supported MLflow artifact
+upload. Recheck their contents; if evidence is unavailable, report incomplete or
+blocked delivery. Do not substitute the legacy launcher's export command.
+
+## Safe evidence and blocked delivery
+
+If recovery remains blocked, preserve a **small, sanitized** evidence bundle in
+an approved durable location before mandated cleanup: invocation/task and job
+IDs, scores/results, evaluated config, coverage and diagnostic summary, relevant
+log excerpts, export error, launcher version, and attempted recovery command
+(with secrets removed). Include any known partial MLflow run IDs and missing
+artifacts. Do not copy caches, checkpoints, or all traces as an export fallback.
+If evidence cannot be preserved, report that blocker before destructive cleanup;
+never silently discard the only results.
+
+In the evaluator handoff, report evaluation and export outcomes independently
+per task: e.g. `evaluation: valid; export: verified (auto/manual)`, `pending`,
+`incomplete`, or `blocked`. Include invocation IDs, verified MLflow run URLs (or
+explicitly none verified), missing evidence, the specific blocker, and absolute
+preserved-evidence paths. A valid evaluation may have blocked delivery; neither
+outcome overrides the other or authorizes another evaluation submission.

plugins/modelopt/skills/evaluation/references/nel-next.md

@@ -199,21 +199,32 @@ with its own `run_id`, copying the shared `services:` block.
   containing `${` — `experiment_name` included, since the same pass resolves the block. Keep `tracking_uri` as `${MLFLOW_TRACKING_URI}` — cross-server, the inherited name
   makes a same-named experiment here and `modelopt_run_url` is the only route back.
 
-## Run (dry-run → canary → full) → push to MLflow
+## Run (dry-run → canary → full) → push → verify MLflow delivery
 
 ```bash
 set -a && source .env && set +a; NEL="$SKILL_DIR/scripts/nel-next.sh"
 "$NEL" eval run <cfg>.yaml --dry-run                                  # validate/render (no SSH)
 "$NEL" eval run <cfg>.yaml --submit -O benchmarks.0.max_problems=2 -O benchmarks.0.repeats=1 -O benchmarks.0.max_concurrent=2   # canary
 "$NEL" eval run <cfg>.yaml --submit                                 # full
 "$NEL" eval {status|logs -f|report -f markdown|merge} -r <run_id>     # lifecycle
-"$NEL" mlflow-push -r <run_id> -c <cfg>.yaml                          # post-run: push merged bundle(s) to MLflow
 ```
 
+Before pushing, apply [Before upload](mlflow-verification.md#before-upload) to
+all outgoing bundles and export metadata. The wrapper fetches the cluster's
+`eval-*.json` unchanged; if they need redaction, use the installed nel-next
+exporter's supported local-bundle path with sanitized copies instead.
+
+```bash
+"$NEL" mlflow-push -r <run_id> -c <cfg>.yaml                          # post-run: push checked merged bundle(s)
+```
+
+Then apply [MLflow delivery verification](mlflow-verification.md), including its
+nel-next evidence-repair path, before handoff or cleanup.
+
 `eval run` on a slurm cluster scp's the sbatch + redacted `.secrets.env` and
 submits via SSH; a built-in afternotok chain auto-resumes across walltime windows;
-sharded runs auto-merge. **SLURM does not auto-export** — `mlflow-push` is the final
-step: it reads the config's `export_config.mlflow`, stages each merged bundle's
+sharded runs auto-merge. **SLURM does not auto-export** — `mlflow-push` reads the
+config's `export_config.mlflow`, stages each merged bundle's
 `eval-*.json` off the cluster (the dev box doesn't mount the run dir), and exports with
 `emit_traces=false` (the default emits one trace per sample → minutes-long hang).
 Idempotent (re-push updates the same run, deduped by `job_id`); forward extra exporter

plugins/modelopt/skills/evaluation/tests/evals.json

@@ -72,12 +72,76 @@
       "Provides SSH-based log monitoring commands for SLURM execution"
     ]
   },
+  {
+    "name": "mlflow-pre-submit-safeguard",
+    "skills": ["evaluation"],
+    "query": "Launch using my existing NEL 0.2.6 config with MLflow auto-export enabled. Generated configs and logs cannot be checked before the CPU exporter uploads them.",
+    "files": [],
+    "expected_behavior": [
+      "Applies the common Step 8 pre-submit safeguard despite skipping config generation; the same gate applies to shortcut-generated configs",
+      "Sets execution.auto_export.destinations to [] before both canary and full submission, retaining MLflow export settings",
+      "Plans checked, sanitized manual delivery from existing results rather than relying on a post-upload scan"
+    ]
+  },
+  {
+    "name": "mlflow-auto-export-verification",
+    "skills": ["evaluation"],
+    "query": "My full evaluation succeeded. The CPU auto-export is still running and MLflow has a same-named canary run. Finish the handoff once export completes.",
+    "files": [],
+    "expected_behavior": [
+      "Checks the separate export job and does not start a competing manual export while it is active",
+      "Finds each full-run MLflow run by invocation and task identity rather than accepting the same-named canary",
+      "Verifies canonical score, evaluated configuration, coverage, and required diagnostic artifact contents against existing results",
+      "Returns verified task-to-run URLs and invocation IDs with separate evaluation and export outcomes, without exporting again"
+    ]
+  },
+  {
+    "name": "mlflow-manual-export-recovery",
+    "skills": ["evaluation"],
+    "query": "Evaluation completed, but the CPU export job failed importing python:3.12.7-slim. One task has a partial MLflow run missing config and diagnostics; another has no run. Recover their exports without rerunning evaluation.",
+    "files": [],
+    "expected_behavior": [
+      "Checks export job state and existing runs by invocation/task identity, rechecking before recovery to avoid duplicates",
+      "Inspects the installed launcher version and CLI help before choosing supported export, image/config, and artifact options",
+      "Repairs or manually exports existing results only; never runs or resumes evaluation to repair export",
+      "Determines whether partial-run repair is supported instead of assuming export idempotency; records any unavoidable replacement run",
+      "Secret-scans and redacts uploaded configs/logs without removing non-secret evaluation provenance",
+      "Verifies scores, configs, and diagnostics in MLflow after recovery; returns actual URLs and invocation IDs with export marked manually verified"
+    ]
+  },
+  {
+    "name": "mlflow-export-blocked-before-cleanup",
+    "skills": ["evaluation"],
+    "query": "Four tasks completed, but all auto-exports failed. Manual export is also blocked by MLflow authentication. Cleanup of the cluster results is due. Report the outcome.",
+    "files": [],
+    "expected_behavior": [
+      "Reports evaluation outcomes separately from blocked or unverified MLflow delivery for all four tasks",
+      "Names the authentication blocker and does not fabricate URLs, infer absent runs from access failure, or claim export success",
+      "Preserves small sanitized results/config/coverage/diagnostic and export-failure evidence with invocation/task IDs in an approved durable location before cleanup",
+      "Reports absolute evidence paths, launcher version and sanitized recovery attempt; flags inability to preserve evidence before destructive cleanup",
+      "Does not upload secrets, copy bulk caches as backup, or rerun evaluation"
+    ]
+  },
+  {
+    "name": "nel-next-explicit-push-verification",
+    "skills": ["evaluation"],
+    "query": "My nel-next SLURM evaluation completed. The merged bundles are safe to upload, but the wrapper stages only eval-*.json; logs remain on the cluster. Deliver verified MLflow results.",
+    "files": [],
+    "expected_behavior": [
+      "Preserves the explicit nel-next.sh mlflow-push -r <run_id> -c <cfg> workflow rather than expecting SLURM auto-export or substituting legacy export syntax",
+      "Checks outgoing bundles and export metadata before pushing and follows the verification gate directly from the nel-next run flow, without relying on excluded Steps 1-9",
+      "Verifies each pushed run by run/bundle job identity, scores, configuration, and required diagnostics before returning actual URLs",
+      "Attaches missing sanitized diagnostic evidence to the identified runs and rechecks contents; does not assume copy_logs=true uploads unstaged logs",
+      "Reports incomplete or blocked delivery if required evidence cannot be delivered, without rerunning evaluation"
+    ]
+  },
   {
     "name": "scicode-eight-run-average",
     "skills": ["evaluation"],
     "query": "Evaluate my NVFP4 checkpoint on the AA suite including SciCode on SLURM",
     "files": [],
     "expected_behavior": [
+      "Applies Step 8's pre-upload safeguard on the AA shortcut path, disabling auto-export before canary and full submission if generated artifacts cannot be checked before upload",
       "Keeps num_repeats at 1 and plans at least 8 independent SciCode submissions instead",
       "Re-submits SciCode alone with -t ns_scicode rather than re-running the whole suite",
       "Reports the SciCode score as the mean of the pooled runs with standard error and run count",