docs: tolerate limited evaluation response truncation

[chadvoegele]

What does this PR do?

Type of change: documentation.

Chad's Agent: distinguish limited output truncation from failed evaluations. A completed Qwen GPQA run scored 90.5619 versus 89.2 with 13/3,168 responses length-capped (0.4104%) at 131,072 tokens; ambiguous guidance prompted an unnecessary rerun.

Use an inclusive, unrounded 1.0% per-benchmark/run tolerance with an explicit response denominator. Within tolerance, warn rather than invalidate or retry, subject to complete sample/scoring coverage and independent validity checks. Above tolerance, return findings and recommendations without automatic resubmission. Verify reference token budgets and deployed context capacity; never tune limits until a score passes or infer negligible score impact from a low rate.

Parent and both evaluator definitions share the policy. Vendored skills remain untouched; the canonical policy explicitly supersedes their conflicting advice. Independent MLflow verification/export gates remain unchanged.

Usage

Guidance only; no API or runtime changes.

Testing

• Changed-file pre-commit hooks passed after rebasing onto origin/main.
• git diff --check origin/main passed.
• Python checks passed: Claude/Codex instruction parity, scenario JSON structure, and threshold arithmetic (0.4104%, 0.99%, 1.0%, 1.01%).
• Added four regression scenarios: reported GPQA case, boundary/denominator accounting, independent failures, and unknown metadata/token-budget mismatches. Agent scenarios were not executed.

Before your PR is "Ready for review"

Contributor and security guidance read; commit signed and signed off.

• Is this change backward compatible?: ✅
• Copied code or new PIP dependency?: N/A
• New necessary tests?: ✅ Regression scenarios added; execution status above.
• Changelog updated?: N/A — agent guidance only.
• Claude approval?: ❌ — not requested; draft.

Additional Information

Draft for review; do not merge. No evaluation jobs submitted.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.25%. Comparing base (cf1f48f) to head (2df014b).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2475      +/-   ##
==========================================
- Coverage   70.91%   70.25%   -0.66%     
==========================================
  Files         600      601       +1     
  Lines       65987    67132    +1145     
==========================================
+ Hits        46794    47166     +372     
- Misses      19193    19966     +773     

Flag	Coverage Δ
unit	58.10% <ø> (-0.09%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[chadvoegele]

Chad's Agent — independent review

Reviewed head: ecda91e5a68dbb1345a5cc492554d197084a643e.

P2 — Make the policy reachable from standalone result analysis

plugins/modelopt/skills/evaluation/references/run-validation.md:41–44 says this policy supersedes launching-evals, but that skill's standalone analysis path never loads it. A fresh session asked to analyze a completed invocation is explicitly routed to launching-evals, whose workflow points to references/analyze-results.md; neither links back to this policy. That reference still requires max_tokens=null for reasoning models and fixed token/context limits. The evaluator role loads both skills, but standalone analysis need not use that role, leaving the conflicting advice active for this supported entry point.

Wire standalone analysis to the canonical policy through a maintained, non-vendored entry point, and add a scenario starting with only launching-evals. All four new scenarios preload both skills, masking this gap.

Validation: Read the entire five-file diff, surrounding evaluation/analysis/comparison guidance, evaluator definitions, and run-gate implementation. Passed git diff --check, the existing agent synchronization/reference/symlink test (invoked directly because pytest was unavailable), scenario JSON/skill-path checks, and exact threshold arithmetic, including 1.0001%. No agent scenarios or live evaluations executed; no checkout changes. No other actionable findings.

[chadvoegele]

Chad's Agent — review follow-up

Addressed the standalone-analysis finding in 2df014b. Moved policy-precedence guidance into maintained AGENTS.md (also exposed through CLAUDE.md), requiring standalone analysis to read canonical run validation before reporting scores. Adapted the existing small-truncation scenario to preload only launching-evals and reject uncapping/fixed-limit advice. Vendored files unchanged.

Passed: changed-file pre-commit hooks, agent-definition synchronization/reference/symlink test, scenario JSON/skill-path and exact-threshold checks, and git diff --check. Fresh-session agent scenario attempted but blocked by expired Claude OAuth; no live evaluations run. Reviewed the diff; pushed without rewriting existing commits. No rebase needed.