docs: fix evaluation output token budget guidance

plugins/modelopt/skills/evaluation/SKILL.md

@@ -309,13 +309,15 @@ nemo_evaluator_config:
       parallelism: ???    # Required — size per references/parallelism.md (bounded by total request count vs GPU serving capacity); ask user in Step 4 if still unclear
       request_timeout: 3600
       max_retries: 10
-      max_new_tokens: 65536  # see rule below
+      max_new_tokens: ???  # resolve from the token-budget rule below
       temperature: 1.0    # from model card (reasoning); adjust
       top_p: 0.95         # from model card (reasoning); adjust
 ```
 
 Per-task `max_new_tokens` overrides are forbidden — set one top-level ceiling everywhere.
 
+**For `max_new_tokens`, the token-budget rule below takes precedence over table recommendations and family fallbacks.**
+
 **Cross-check `temperature` / `top_p` / `max_new_tokens` against `references/nvfp4-modelcard-sampling.md`** — the published settings for the 2026 NVFP4 checkpoints under `huggingface.co/nvidia` that disclose them (older releases and cards that publish nothing are absent — for those, read the card; `-DSpark` / `-DFlash` spec-decode variants share their base checkpoint's row, since spec decoding does not change the target's output distribution). **The card is the source of truth; this file is a reference, not a constraint** — use it to confirm a value you read, to fill a gap when the card is silent or ambiguous, and to catch a misreading. Worth consulting whenever the model is an NVFP4 checkpoint **or shares a family with one** (Qwen3.x, GLM-4.7/5.x, Kimi K2.x/K3, MiniMax M2.x/M3, DeepSeek V3.x/V4/R1, Gemma 4, Nemotron 3/3.5, Llama-Nemotron, Mistral Medium 3.5), and especially when you are unsure. It is a dated snapshot, so for anything newer than it, trust the card. See that file's "Lookup" section.
 
 **`temperature` / `top_p` are different: per-task overrides ARE allowed and often required.** Cards often specify sampling per scenario — DeepSeek-V4-Pro-0813 gives `top_p = 0.95` for agentic scenarios and `1.0` otherwise, so a single top-level `0.95` is wrong for every non-agentic task.
@@ -326,13 +328,10 @@ overridden task is reported under sampling params it did not use.
 
 #### `max_new_tokens` — mandatory model-card lookup
 
-1. **Fetch the HF model card before writing the value.** Not optional.
-2. Scan for any `max_tokens` / `max_new_tokens` / "output length" recommendation. Pick the **highest** value the card mentions (Qwen3.6: 32768 general + 81920 math-coding → use **81920**). Annotate with a citing comment.
-   **Card figures are SINGLE-TURN.** On multi-turn / agentic benchmarks the model's own answer is fed back in, so the cap must satisfy `n_turns × max_new_tokens + prompt < max_model_len`. Taking a card's headline "384K output" literally lost SciCode samples to HTTP 400; 65536 was clean. (`references/run-validation.md` already covers checking `finish_reason: length` after a run.)
-3. **Consult `references/nvfp4-modelcard-sampling.md` as a reference.** Listed and in agreement → proceed with confidence. Listed and different → **the card wins**; re-read it, then note the discrepancy for the user rather than auto-correcting either way. Not listed, or the card is silent or ambiguous → take the nearest same-family rows as the value, a far better prior than the generic fallback below. Its `max_num_tokens` column records the card's *headline* cap, so rule 2 above still governs: when a card names more than one cap, the highest wins even if that exceeds the row.
-4. If the card is genuinely silent after a thorough read **and** the family table offers no usable pattern, fall back to: **65536** (reasoning), **16384** (non-reasoning); surface the silence to the user.
-5. **Forbidden:** writing `max_new_tokens: <generic_default>` with a "card not yet checked" comment. Either fetch and apply, or fetch and confirm silence.
-6. **A higher cap doesn't fix runaway reasoning.** On hard tasks (e.g. HLE) a non-terminating model just rambles to the larger cap (~80% length-capped at 131072), and the cap only helps if deployment `--max-model-len > prompt + max_new_tokens` (else generation is silently clipped — AA-LCR's ~120K input leaves little room). Treat such tasks as low-confidence.
+1. **Read the HF model card before setting the value.** Identify the model's reasoning mode, creator-disclosed maximum output length, and any `max_tokens` / `max_new_tokens` used for the applicable evaluation. Cite the source and rationale in a config comment; quickstart examples are not evaluation budgets.
+2. **Non-reasoning:** use **16384**, lowered for smaller output caps or available context after input tokens. **Reasoning:** use the maximum output length allowed and disclosed by the model creators. Do not substitute 65536, a same-family budget, or the context-window size for an undisclosed output maximum.
+3. **An explicit model-card budget used for the applicable evaluation can override these defaults.** Do not apply another benchmark's budget or choose the highest number mentioned. Keep one top-level `max_new_tokens` (no per-task overrides); conflicting benchmark-specific budgets require separate configs.
+4. Verify that the output budget fits alongside the input, including accumulated multi-turn history. Surface conflicts rather than silently clipping a creator/evaluation budget. Check `finish_reason: length` after the run (`references/run-validation.md`); a higher cap does not fix runaway reasoning.
 
 #### Quantization-aware benchmark defaults
 

plugins/modelopt/skills/evaluation/recipes/examples/example_eval.yaml

@@ -132,7 +132,7 @@ evaluation:
         parallelism: ???  # Number of concurrent requests per each benchmark
         request_timeout: 3600
         max_retries: 10
-        max_new_tokens: 65536  # 64K for reasoning models; use 16384 (16K) for non-reasoning; prefer model card value
+        max_new_tokens: ???  # Resolve per SKILL.md's token-budget rule; match the export description/tag below
         temperature: 1.0       # from model card (reasoning mode); adjust per card
         top_p: 0.95            # from model card (reasoning mode); adjust per card
     target:
@@ -197,7 +197,7 @@ export:
     # would resolve. Cross-server, that makes a same-named experiment here and only
     # modelopt_run_url reaches the PTQ run (Step 4).
     experiment_name: ${oc.env:USER}/CHANGEME-served-model-name
-    description: 'CHANGEME-served-model-name | T=1.0, top_p=0.95, max_new_tokens=65536'
+    description: 'CHANGEME-served-model-name | T=1.0, top_p=0.95, max_new_tokens=???'
     log_logs: true
     log_artifacts: true
     only_required: false
@@ -207,7 +207,7 @@ export:
       model: CHANGEME-served-model-name
       temperature: '1.0'
       top_p: '0.95'
-      max_new_tokens: '65536'
+      max_new_tokens: ???
       # From the checkpoint's .experiment.json when present; drop when absent. Keep the
       # quotes (a bare 20260910 becomes a date) and skip any value containing ${ -- quoting
       # does not stop OmegaConf resolving it.

plugins/modelopt/skills/evaluation/references/model-card-research.md

@@ -12,21 +12,19 @@ Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it ca
     `temperature` / `top_p` / `max_num_tokens` for the 2026 NVFP4 checkpoints
     under `huggingface.co/nvidia` that disclose them, grouped by family.
     Required for any NVFP4 checkpoint or same-family sibling; it is also the
-    best source of a default when the card is silent. Pre-2026 releases are out
+    best source of a sampling default when the card is silent; output budgets
+    follow the rule below instead. Pre-2026 releases are out
     of scope there — read their cards.
 - Context length (`deployment.extra_args: "--max-model-len <value>"`)
-- **Output length (`max_new_tokens`) — mandatory extraction.** Scan the
-  card for any `max_tokens` / `max_new_tokens` / "output length"
-  recommendation. Cards often list two values (e.g., Qwen3.x: `32768`
-  thinking-general + `81920` math/coding). **Pick the highest value** and
-  apply at the top level (no per-task overrides). If the card is genuinely
-  silent on output length, note that explicitly and fall back to the
-  generic default (64K reasoning / 16K non-reasoning) — never write a
-  config with "card not yet checked" + generic default. Check
-  `nvfp4-modelcard-sampling.md` for the model or its family before falling
-  back; a same-family published cap beats the generic default. See SKILL.md
-  Step 3 "`max_new_tokens` — pick a single top-level value" for the full
-  rule.
+- **Output length (`max_new_tokens`) — mandatory extraction.** Record the
+  creator-disclosed maximum output length and any budget used for the applicable
+  evaluation, with sources. Non-reasoning defaults to **16384**, lowered for
+  smaller context/output caps; reasoning uses the creator-disclosed output
+  maximum. An explicit model-card budget for the applicable
+  evaluation can override these defaults. No highest-number or same-family
+  fallback. Keep one top-level value; conflicting benchmark budgets require
+  separate configs. See SKILL.md Step 3
+  "`max_new_tokens` — mandatory model-card lookup" for the full rule.
 - TP/DP settings (to set them appropriately, AskUserQuestion on how many GPUs the model will be deployed)
 - Reasoning config (if applicable):
   - reasoning on/off: use either:

plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md

@@ -15,6 +15,10 @@ is silent or ambiguous. It does not replace reading the card — see
 
 ## Lookup
 
+**The lookup and family priors below govern `temperature` / `top_p`, not output
+budgets.** For `max_new_tokens`, follow SKILL.md Step 3's token-budget rule;
+verify table values against the card and the applicable evaluation.
+
 **The card is the source of truth; this table is a reference, not a constraint.**
 Use it to confirm what you read, to fill a gap when the card is silent, and as a
 sanity check when you are unsure — never to override a value the card states.
@@ -23,16 +27,11 @@ sanity check when you are unsure — never to override a value the card states.
    it, but note in the config comment that it is recommended sampling, not a
    stated eval setting; if a same-family `eval` row disagrees, surface both.
    `—` → that field is unpublished; resolve **it alone** via step 2.
-   `max_num_tokens` is the card's *headline* cap — where a note names a higher
-   per-task cap (GLM-5.2 GPQA `100000`, Qwen3.5-397B-V2 τ²-Telecom `128000`,
-   Kimi-K3 uncapped for Terminal-Bench) and that task is in your suite, SKILL.md
-   Step 3's take-the-highest rule governs the single top-level value, not the
-   column.
 2. **No row** (new or unreleased variant, non-NVIDIA baseline, pre-2026) → take
    the nearest same-family rows as the expected value.
 3. **Card vs. table.** Agree → proceed. Card silent + family consistent → adopt
-   the family value and cite this file in a line comment; that beats SKILL.md
-   Step 3's generic 65536 / 16384. **Card disagrees → the card wins**, but
+   the sampling value and cite this file in a line comment.
+   **Card disagrees → the card wins**, but
    surface it — defaults shift between generations, so a mismatch means re-read,
    not auto-correct.
 4. **Baseline and candidate share one setting.** Cards report both precisions
@@ -104,9 +103,9 @@ that tie. `max_num_tokens` is the max generation length, i.e.
   `384000` is a long-context outlier.
 - **Per-task overrides are narrow** — SciCode (lower temperature), τ²-Bench
   Telecom (greedy or larger cap), GPQA Diamond (larger cap), Terminal-Bench
-  (uncapped). SKILL.md Step 3 forbids per-task `max_new_tokens`, so when a card
-  lists two caps **take the maximum** as the single top-level value and note the
-  split in a comment.
+  (uncapped). SKILL.md Step 3 forbids per-task `max_new_tokens`; conflicting
+  benchmark-specific budgets require separate configs,
+  not taking the maximum.
 
 ## Refreshing
 

plugins/modelopt/skills/launching-evals/references/analyze-results.md

@@ -44,9 +44,9 @@ Check logs for silent errors that may invalidate results:
 
 1. **Methodology consistency**: Verify same benchmark versions, prompt templates, sampling params, and infrastructure across all models. Flag discrepancies.
 2. **HF model card compliance**: Read the model's HuggingFace model card. Flag any deviations in inference parameters (temperature, top_p, max_new_tokens, deployment args, reasoning flags, etc.).
-3. **Reasoning model validation**: Verify temp > 0, top_p > 0, `max_tokens` = null (allow full output length).  
+3. **Reasoning model validation**: Verify temp > 0, top_p > 0; `max_tokens` / `max_new_tokens` uses the creator-disclosed maximum output length, unless the model card specifies a budget used for the applicable evaluation (see `../../evaluation/SKILL.md`, Step 3).
    NOTE: `use_reasoning: False` in adapter_config does NOT mean reasoning is disabled — it only controls the reasoning interceptor. Whether reasoning is active depends on the model's own controls (deployment args, system prompt, API payload fields, etc.).
-4. **Non-reasoning model validation**: Verify `max_tokens` = 16k
+4. **Non-reasoning model validation**: Verify `max_tokens` / `max_new_tokens` = 16384, lowered for smaller context/output caps, unless the model card specifies a budget used for the applicable evaluation.
 5. **Max model length**: Verify `max-model-len` = 131072 (leaderboard-recommended). Long context benchmarks (AA LCR, RULER) and agentic benchmarks may require a longer `max-model-len`.
 6. **RULER tasks**: Check thinking disabled, walltime=4h, rope-scaling for Qwen models
 7. **AA baseline comparison**: Compare results against Artificial Analysis published scores. Exact match not expected — flag significant deviations.