Skip to content

docs: require parent authorization for evaluation resubmissions - #2473

Draft
chadvoegele wants to merge 1 commit into
NVIDIA:mainfrom
chadvoegele:docs/eval-parent-authorization
Draft

chadvoegele wants to merge 1 commit into
NVIDIA:mainfrom
chadvoegele:docs/eval-parent-authorization

Conversation

@chadvoegele

Copy link
Copy Markdown
Contributor

What does this PR do?

Type of change: documentation.

Chad's Agent: Require explicit parent authorization for each delegated evaluation submission. An evaluator could previously submit a speculative retry while the parent accepted the completed result and stopped.

Reuse the existing handoff/reply mechanism: report terminal results, then wait. Keep score and output-health acceptance aligned with the evaluation skill, preserve failed/canceled invocations and resume-job lineage, and verify cancellation of actual submitters and affected jobs rather than wrappers. No orchestration code added.

Usage

Parent reviews the terminal report and remaining budget, then says STOP or authorizes one task/scope/configuration/budget. Silence does not authorize another submission.

Testing

  • Agent-definition synchronization/reference test: 1 passed (uv run --no-project --with pytest --with pytest-timeout python -m pytest -o addopts='' plugins/modelopt/skills/day0-release/tests/test_agent_definitions.py -q).
  • Pre-commit checks on all four changed files passed.
  • git diff --check passed.
  • Added three regression scenarios covering GPQA stop/silence, one-submission authorization, and cancellation/resume accounting. These are behavioral examples, not executed agent tests; no evaluations submitted.

Before your PR is "Ready for review"

  • Contributor and security guidance reviewed; commit signed and signed off.
  • Is this change backward compatible?: N/A — agent instructions intentionally tighten submission authorization; no API changes.
  • If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: N/A.
  • Did you write any new necessary tests?: ✅ — regression scenarios; existing cross-agent parity test passes.
  • Did you update Changelog?: N/A — instruction-only change.
  • Did you get Claude approval on this PR?: ❌ — draft, not requested.

Additional Information

Draft for review; do not merge.

Signed-off-by: Chad Voegele <cvoegele@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 18, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Sep 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.91%. Comparing base (cf1f48f) to head (1b06cd4).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2473   +/-   ##
=======================================
  Coverage   70.91%   70.91%           
=======================================
  Files         600      600           
  Lines       65987    65987           
=======================================
  Hits        46794    46794           
  Misses      19193    19193           
Flag Coverage Δ
unit 58.19% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@chadvoegele

Copy link
Copy Markdown
Contributor Author

Chad's Agent — independent review

Reviewed head 1b06cd4c39799d1cbbe5cbacb046bd3756cc7a12.

No actionable findings. The delegated-evaluation rules explicitly gate subsequent submissions, preserve resume lineage, and keep score acceptance subordinate to run validation.

Checks: read the entire four-file diff and surrounding evaluation, validation, monitoring, workspace, and SciCode instructions; inspected NEL's SLURM resume/cancellation implementation. Agent-definition synchronization/reference test: 1 passed. Scenario JSON, fixture paths, and git diff --check: passed.

Limits: the three new scenarios are behavioral examples, not executed agent tests. No evaluations or cluster cancellation tests were run; this review does not establish runtime enforcement of authorization or budgets. Shared checkout unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant