diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index 55178d1fbf0..5dddbf80b2d 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -42,7 +42,7 @@ for one, do **not** add it to a 0.2.6 `evaluation.tasks` list — instead: 1. Read **`references/nel-next.md`** (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe `recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md`; start from `recipes/examples/example_eval_next.yaml`. 2. Isolated nel-next venv: `"$SKILL_DIR/scripts/nel-next.sh" --setup-only` (keeps 0.2.6 `nel` untouched). -3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`) into `.env`; always include the `output.export_config.mlflow` block. +3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`, `${HARBOR_ECS_REGION}`) into `.env`; always include the `output.export_config.mlflow` block. 4. Dry-run → canary → full (`nel-next.sh eval run`), then **push to MLflow** — SLURM doesn't auto-export, so run `nel-next.sh mlflow-push -r -c ` after (config-driven; see `references/nel-next.md`). Steps 1–9 below are currently validated with 0.2.6 — use them for everything else. @@ -64,6 +64,8 @@ for an "AA" request. If the user asks for MRCR: 2. **Pick the variant first** (`config_n3_1m` / `config_n3_128k` / `config`) — it sets the context cap, dataset *and* metric prefix; the three are not comparable; set it in **both** `data_prep_params` and `collect_rollout_params`. + Upstream now carries 1M and 128K as two separate benchmarks with their own + manifests and metric keys — see the recipe. 3. `.env`: `HF_TOKEN` (dataset + n3 tokenizer are gated) plus `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the `pre_cmd` installs `tiktoken` + `transformers`; prepare fails without it) and diff --git a/plugins/modelopt/skills/evaluation/recipes/env.example b/plugins/modelopt/skills/evaluation/recipes/env.example index bb742fcf60f..a02fee7c6e0 100644 --- a/plugins/modelopt/skills/evaluation/recipes/env.example +++ b/plugins/modelopt/skills/evaluation/recipes/env.example @@ -54,6 +54,9 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1 # AWS_ACCESS_KEY_ID= # AWS_SECRET_ACCESS_KEY= # NEL_NEXT_EVAL_IMAGE= # harbor eval image (arch-matched) — set by eval-config +# HARBOR_ECS_REGION= # Fargate sandbox region. Upstream defaults: us-east-1 +# # (Terminal-Bench), us-east-2 (SWE-bench). TB's ECR repo +# # NAME tracks this (harbor-); SWE-bench's does not. # HARBOR_ECR_REPOSITORY= # Terminal-Bench harbor ECR — set by eval-config -# HARBOR_SWEBENCH_ECR_REPOSITORY= # SWE-bench harbor ECR (us-west-2) — set by eval-config +# HARBOR_SWEBENCH_ECR_REPOSITORY= # SWE-bench harbor ECR (us-west-2, region-independent) — set by eval-config # MLFLOW_TRACKING_URI= # MLflow tracking URI (canonical frontier-evals host) — set by eval-config diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml index 9463c653062..1f7850f8782 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -45,17 +45,27 @@ services: proxy: request_timeout: 3600 # canonical; MUST be >= benchmarks[].solver.agent_kwargs.llm_kwargs.timeout extra_body: {skip_special_tokens: false} # add model-card sampling extras here if the card specifies them; mirror them in the export tags below + model_traffic: {capture_request_body: true} # FEA-224 — set PER SERVICE, never in a shared block (an alias-only entry has no `type` and fails the service discriminator) + # Interceptor lists REPLACE wholesale on merge — if you add one, restate them all. interceptors: + # FEP-1104/1120 diagnostics — canonical position is FIRST (both TB2.1 and SWE-bench). + # Uncomment for a CANARY/debug run, drop it for the scored run: first_n caps only 200s, + # so every error pair (full req+res bodies) is retained in memory for the whole run and + # re-serialized on each write — unbounded growth exactly when the server errors. + # - name: http_pairs_dump + # config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50} # $$ defers expansion to run time + # SWE-bench (OpenHands, multi-turn) inserts system_message + turn_counter HERE, before + # drop_params — don't lift that chain blind, see swebench_verified.md - name: drop_params # agents send max_tokens; many servers reject it # last two are sent by the 0.5.x harbor eval image; vLLM 400s on them unless stripped config: {params: [max_tokens, max_completion_tokens, max_input_tokens_per_task, no_rebuild]} - # SWE-bench (OpenHands, multi-turn) adds turn_counter + system_message AND USES A DIFFERENT - # ORDER (drop_params before consolidate_system) — don't lift this chain — see swebench_verified.md - # FEP-1104/1120 diagnostics — uncomment for a CANARY/debug run, drop it for the scored run: - # first_n caps only 200s, so every error pair (full req+res bodies) is retained in memory for - # the whole run and re-serialized on each write — unbounded growth exactly when the server errors. - # - name: http_pairs_dump # canonical LAST in the chain (SWE-bench: first) - # config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50} # $$ defers expansion to run time + - name: consolidate_system + # reasoning models only — normalize the reasoning field and replay it across turns. + # `reasoning_replay.mode` is per MODEL (think_tags default / native / both), not per + # benchmark. Drop both for instruct models. + # - name: reasoning + # - name: reasoning_replay + # config: {mode: think_tags} node_pool: gpu benchmarks: @@ -67,10 +77,12 @@ benchmarks: timeout_strategy: max # canonical TB2.1; "task" = leaderboard-comparable agent_kwargs: {llm_kwargs: {timeout: 3600}} sandbox: - region: us-east-1 # MUST match the region in ${HARBOR_ECR_REPOSITORY} (SWE-bench: us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY}) + # TB2.1's ECR repo NAME tracks the region (harbor-); SWE-bench's is pinned to + # us-west-2/harbor-swebench whatever the sandbox region is (us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY}). + region: ${HARBOR_ECS_REGION:-us-east-1} ecr_repository: ${HARBOR_ECR_REPOSITORY} # from modelopttools:eval-config concurrency: 50 - log_stream_prefix: terminalbench21-??? + log_stream_prefix: terminalbench-21-??? # canonical stem + - cluster: type: slurm @@ -87,7 +99,7 @@ cluster: HF_HOME: /cache/huggingface AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} - AWS_DEFAULT_REGION: us-east-1 # match sandbox.region / the region in ${HARBOR_ECR_REPOSITORY} + AWS_DEFAULT_REGION: ${HARBOR_ECS_REGION:-us-east-1} # match sandbox.region / the region in ${HARBOR_ECR_REPOSITORY} LLM_API_KEY: "no-key-needed" mount_home: false auto_resume: true @@ -110,4 +122,12 @@ output: exclude_patterns: ["shard*", "model_traffic.jsonl"] # captured request bodies (FEA-224) stay in the run dir description: ??? # ' | T=1.0 top_p=0.95 | (timeout_strategy=…) | r8' # model/checkpoint_path/benchmark drive dashboard attribution (engine logs only a generic metric key); temperature/top_p mirror generation above. - tags: {framework: vllm, model: "???", checkpoint_path: "???", benchmark: "???", temperature: '1.0', top_p: '0.95'} + tags: + framework: vllm + model: "???" + checkpoint_path: "???" + benchmark: "???" + task_name: "terminal-bench-2.1" # upstream benchmark slug (SWE-bench: swebench-verified) + nemo-evaluator-next-version: "0.5.0.1" # eval_image version, so a score traces to its image + temperature: '1.0' + top_p: '0.95' diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml index 2f84bbf8118..e1d8da46d19 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml @@ -86,9 +86,11 @@ deployment: gpu_memory_utilization: 0.95 # golden raises this from 0.85 to fit the 1M KV cache # Golden serves in REASONING mode (thinking forced on via chat_template_kwargs # below), so set the model's --reasoning-parser. - # After sizing num_nodes / num_instances / TP / DP above, append - # `--max-num-seqs N` where N = ceil(parallelism / num_instances / DP) — the GYM - # rule, not the generic SKILL one. Golden: ceil(256/4/2) = 32. + # After sizing num_nodes / num_instances / TP / DP above, append `--max-num-seqs N`. + # Do NOT derive N from `parallelism`: the canonical bench.yaml now sets parallelism + # 512 DELIBERATELY ABOVE total server capacity so 1M-token requests queue at the + # endpoint instead of leaving replicas idle. Size N from what a replica's KV can hold + # (the golden ran 32 per replica × 8 replicas = 256 in flight against parallelism 512). # Treat that as a CEILING, not a target. Like AA-LCR (recipes/tasks/aa/lcr.md) # this task is KV-bound, and far more so: ~1M input tokens per request vs LCR's # ~120K. Too high and vLLM preempts, and recomputing a 1M-token prefill makes the @@ -131,10 +133,16 @@ evaluation: params: temperature: 1.0 top_p: 0.95 - # Gym client concurrency, TOTAL across instances — not a per-server cap. - parallelism: 256 + # Gym client concurrency, TOTAL across instances — not a per-server cap, and + # intentionally ABOVE it: the canonical bench.yaml uses 512 so the endpoint + # always has queued work while Gym's own dispatch stays bounded. Drives both + # ++num_samples_in_parallel and ++global_aiohttp_connector_limit_per_host. + parallelism: 512 request_timeout: 36000 # 1M-token prefills are slow max_retries: 10 + # Empty (null) = full run. Set N for a canary: MRCR forwards it to the gym + # as ++limit (see the gated line in collect_rollout_params below). + limit_samples: # Uncapped: answers reproduce a full earlier turn, and any cap truncates # it and craters SequenceMatcher.ratio (see max_output_tokens=null below). max_new_tokens: @@ -292,9 +300,12 @@ evaluation: ++reuse_existing_data_preparation=true ++skip_venv_if_present=true ++upload_rollouts_to_wandb=false + ++observability_enabled=true + ++model_call_capture_dir={{config.output_dir}}/model_calls ++responses_create_params.temperature={{config.params.temperature}} ++responses_create_params.top_p={{config.params.top_p}} ++responses_create_params.max_output_tokens=null + {% if config.params.limit_samples is not none %}++limit={{config.params.limit_samples}}{% endif %} common_params: >- ++num_repeats=1 ++policy_base_url={{target.api_endpoint.url}} diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md index bbd70551cb9..289cf7c672d 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/swebench_verified.md @@ -8,6 +8,11 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. > **Source of truth:** `configs/benchmarks/swe-bench-verified/bench.yaml` in > nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the eval-image > pin in `configs/shared/nel_next_containers.yaml` — match its values for a reference run. +> +> That dir also ships `bench_direct.yaml`, a **different backend**: the Gym/opencode +> route (`++num_samples_in_parallel=1024`, its own Gym commit pin, opensandbox) used by +> the P/D recipes. `benchmark_backend: evaluator` — the harbor path described here — is +> what this skill runs. Don't mix values between the two. ## Task-specific values (canonical `bench.yaml`) @@ -16,12 +21,15 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. | `playbook` | `swebench_verified` (`harbor://swebench-verified@1.0`) | | agent | `openhands-sdk` (playbook; `agent_kwargs: {max_iterations: 200, version: "1.17.0"}`) | | scope | 500 Python tasks × `repeats: 5` | -| `max_concurrent` / `sandbox.concurrency` | `15` in `bench.yaml`; per-model configs override it (MiniMax-M2.7 uses `20`) | +| `max_concurrent` / `sandbox.concurrency` | `15` (canonical `bench.yaml`); the reviewed per-model leaves keep it. A separate customer-launch track (`reviewed_configs/clu/…`) runs `repeats: 3` / `concurrency: 10` — **not** AA-comparable, don't borrow its numbers | | `solver` | `timeout_strategy: max`, `run_timeout: 10800` (3h), `agent_kwargs.llm_kwargs.timeout: 3600` | -| `sandbox.region` | `us-east-2` | -| `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` (dedicated `harbor-swebench` repo, **us-west-2**, regardless of sandbox region) | +| `sandbox.region` | `${HARBOR_ECS_REGION:-us-east-2}` (differs from TB2.1's `us-east-1` default) | +| `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` → `463701203462.dkr.ecr.us-west-2.amazonaws.com/harbor-swebench`. **Hardcoded us-west-2 upstream** — unlike TB2.1 the repo does *not* track `HARBOR_ECS_REGION`; moving the sandbox region leaves the ECR alone | | `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` → **`0.5.0.1-harbor`** (same pin as TB2.1: `configs/shared/nel_next_containers.yaml`) *(shared — see `references/nel-next.md`)* | -| `cluster.container_env.AWS_DEFAULT_REGION` | `us-east-2` (match `sandbox.region`) | +| `cluster.container_env.AWS_DEFAULT_REGION` | `${HARBOR_ECS_REGION:-us-east-2}` (match `sandbox.region`) | +| `sandbox.log_stream_prefix` | canonical stem `swebench---` | +| `output.export_config.mlflow.tags` | canonical adds `task_name: swebench-verified` and `nemo-evaluator-next-version: 0.5.0.1` | +| `proxy.model_traffic.capture_request_body` | `true`, set **per service** (`services..proxy.model_traffic`) — never in a shared block | | `instruction_template` | `/configs/prompts/swebench_instruction.md`, **must be MOUNTED**; content is scoring-relevant (gotcha below) | | `proxy.request_timeout` | `3600` (FEP-1104 paired HTTP timeout; leaves mirror it on the service proxy) *(shared — see `references/nel-next.md`)* | | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` *(shared — see `references/nel-next.md`)* | @@ -32,7 +40,7 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`. benchmarks: - playbook: swebench_verified repeats: 5 - max_concurrent: 15 # keep == sandbox.concurrency; per-model configs may raise both + max_concurrent: 15 # canonical; keep == sandbox.concurrency instruction_template: /configs/prompts/swebench_instruction.md # mounted (see gotcha) solver: service: @@ -40,7 +48,7 @@ benchmarks: run_timeout: 10800 agent_kwargs: {llm_kwargs: {timeout: 3600}} sandbox: - region: us-east-2 + region: ${HARBOR_ECS_REGION:-us-east-2} # canonical; the ECR below stays us-west-2 ecr_repository: ${HARBOR_SWEBENCH_ECR_REPOSITORY} concurrency: 15 log_stream_prefix: swebench-verified-- @@ -88,10 +96,16 @@ OpenHands runs ~200 turns/task. The canonical config adds a `system_message` interceptor (a large OpenHands system prompt — copy it verbatim from `bench.yaml`) plus `turn_counter`. -**Order differs from TB2.1**: `http_pairs_dump` is **first** (not last) and `drop_params` -comes **before** `consolidate_system`. `http_pairs_dump` is canary/diagnostic-only — it -retains every error pair in memory for the whole run (`references/nel-next.md`); drop it -from the scored config. +**Order.** `http_pairs_dump` is **first**, then `system_message` → `turn_counter` → +`drop_params` → `consolidate_system` → `reasoning` → `reasoning_replay`. (TB2.1 now puts +`http_pairs_dump` first too; the SWE-bench-only part is the `system_message` + +`turn_counter` pair.) `http_pairs_dump` is canary/diagnostic-only — it retains every error +pair in memory for the whole run (`references/nel-next.md`); drop it from the scored config. + +**Interceptor lists replace wholesale on merge.** The reviewed leaf restates the model +fragment's whole chain to add two entries — it does not append. Adding `system_message` +without re-listing `drop_params`/`consolidate_system`/`reasoning`/`reasoning_replay` +silently drops them. ```yaml proxy: @@ -127,11 +141,21 @@ Fargate quota and `N × gpus_per_node` against your allocation. ## Score Extraction -Report **`pass@1`** only — benchmark `swebench-verified@1.0`, scorer `pass@1` (0–1): +Report **`pass@1`** — benchmark `swebench-verified@1.0`, scorer `pass@1` (0–1): the resolved rate over the 500 tasks, **already averaged over repeats** (nel-next reports a single `pass@1`; there is **no `avg-of-N` key** like the 0.2.6 nemo-skills -metrics). MLflow logs it as `pass_at_1`. Read from `report.md` (Benchmark / Scorer -table) in the run dir or `nel eval report -r `, then push to MLflow with +metrics). MLflow logs it as `pass_at_1`. + +> The upstream `manifest.yaml` lists its metric key as **`mean/reward`**, not `pass@1` — +> that is the key the Gym/direct (`bench_direct.yaml`) recipes emit, and it is shared by +> the whole SWE-bench family (Verified / Multilingual / Pro). On the harbor path the +> certificates in `reviewed_configs/…/swebench_verified/certificates/` record `pass@1` +> (e.g. `0.516`), so `pass@1` is still what you report for a nel-next run. Quote the key +> name alongside the number so a `mean/reward` from the direct path is never compared +> against it. + +Read from `report.md` (Benchmark / Scorer table) in the run dir or +`nel eval report -r `, then push to MLflow with `nel-next.sh mlflow-push -r -c ` (SLURM doesn't auto-export). Keep `timeout_strategy` + the instruction/system prompt fixed across baseline vs quantized for a valid delta. diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md index 42a5366735a..b022d74c498 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md @@ -18,23 +18,35 @@ pinned via a vendored registry override shipped in the `nemo-evaluator` package. | agent | `terminus-2` (from the playbook) | | `repeats` | `8` (AA / leaderboard count — don't lower for a scored run) | | sandbox | `ecs_fargate`, `stateful: true` (agent + verifier share one container) | -| `sandbox.region` | match the region in `${HARBOR_ECR_REPOSITORY}` (us-east-1 with the eval-config default) | -| `sandbox.ecr_repository` | `${HARBOR_ECR_REPOSITORY}` — set by the `modelopttools:eval-config` skill (internal harbor ECR) | +| `sandbox.region` | `${HARBOR_ECS_REGION:-us-east-1}` — one env var now drives region **and** repo name | +| `sandbox.ecr_repository` | `${HARBOR_ECR_REPOSITORY}` — set by `modelopttools:eval-config`; canonical value is `463701203462.dkr.ecr..amazonaws.com/harbor-`, i.e. the repo **name tracks the region** (`harbor-us-east-1`). Changing region means changing both halves. | | `cluster.container_env.AWS_DEFAULT_REGION` | match `sandbox.region` | +| `sandbox.log_stream_prefix` | canonical stem `terminalbench-21`; the reviewed leaf appends model + framework (`terminalbench-21-nano-v35-vllm`) | | `max_concurrent` / `sandbox.concurrency` | `50` (canonical bench.yaml) | | timeout_strategy | `max` (canonical bench.yaml) + `agent_kwargs.llm_kwargs.timeout: 3600`; use `task` for leaderboard-comparable | | `cluster.eval_image` | **`0.5.0.1-harbor`** (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch) *(shared — see `references/nel-next.md`)* | | `proxy.request_timeout` | `3600` — must be **≥** `agent_kwargs.llm_kwargs.timeout` *(shared — see `references/nel-next.md`)* | | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` *(shared — see `references/nel-next.md`)* | | `output.export_config.mlflow.exclude_patterns` | `["shard*", "model_traffic.jsonl"]` *(shared — see `references/nel-next.md`)* | -| `http_pairs_dump` | **last** in the interceptor chain — canary/diagnostic only, drop it for a scored run (unbounded error-pair retention) | +| `http_pairs_dump` | **first** in the interceptor chain (same position as SWE-bench) — canary/diagnostic only, drop it for a scored run (unbounded error-pair retention) | +| `proxy.model_traffic.capture_request_body` | `true`, set **per service** (`services..proxy.model_traffic`) — never in a shared block, which would inject an alias-only entry with no `type` into other models' composed configs | +| `output.export_config.mlflow.tags` | canonical adds `task_name: terminal-bench-2.1` and `nemo-evaluator-next-version: 0.5.0.1` alongside `model`/`framework` | | scope | 89 tasks × `repeats: 8` | These values mirror the canonical TB2.1 config — re-check it before a scored run: `configs/benchmarks/terminal-bench-2.1/bench.yaml` (+ `manifest.yaml`) in nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the image pin in `configs/shared/nel_next_containers.yaml`. See `references/nel-next.md` + the eval-config -"source of truth" note. The `benchmarks:` block (drop into the example template): +"source of truth" note. + +> **`bench_direct.yaml` is a different backend, not an alternative spelling.** The same +> upstream dir now also ships a Gym-native route (Terminus 2 agent + opensandbox, +> `++num_samples_in_parallel=512`, its own Gym commit pin) used by the P/D recipes. It +> scores the same 2.1 task set but through different machinery; `benchmark_backend: +> evaluator` — the harbor path below — is what this skill runs and what the nel-next +> deltas here describe. Don't mix values between the two. + +The `benchmarks:` block (drop into the example template): ```yaml benchmarks: @@ -49,10 +61,10 @@ benchmarks: llm_kwargs: timeout: 3600 # per-request LLM timeout (canonical) sandbox: - region: us-east-1 # must match the region in ${HARBOR_ECR_REPOSITORY} + region: ${HARBOR_ECS_REGION:-us-east-1} # canonical; repo name below must track it ecr_repository: ${HARBOR_ECR_REPOSITORY} # from eval-config (internal harbor account/region) concurrency: 50 - log_stream_prefix: terminalbench21-- + log_stream_prefix: terminalbench-21-- ``` `cluster.eval_image: ${NEL_NEXT_EVAL_IMAGE}` (`0.5.0.1-harbor`) and the AWS creds @@ -66,7 +78,8 @@ it is purely a wall-clock lever. `shards: 4` suits 89 × r8 = 712 trials. Check ## Score Extraction -Report **`pass@1`** only — benchmark `terminal-bench@2.1`, scorer `pass@1` (0–1): +Report **`pass@1`** — the key the upstream `manifest.yaml` certifies on, and the only +one to quote. Benchmark `terminal-bench@2.1`, scorer `pass@1` (0–1): the resolved rate over the 2.1 task set, **already averaged over repeats** (a single `pass@1`; no `avg-of-N` key). MLflow logs it as `pass_at_1`. Read from `report.md` (Benchmark / Scorer table) or `nel eval report -r `, then push to MLflow with diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md b/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md index 2e32390b975..27a3d3c1147 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/gym/mrcr.md @@ -5,6 +5,9 @@ - Benchmark: - Resource server: - Dataset: `openai/mrcr` (HF, gated → `HF_TOKEN`) +- **Source of truth:** `configs/benchmarks/mrcr/bench.yaml` + `manifest.yaml` (1M) and + `configs/benchmarks/mrcr-128k/` (128K) in nvidia-eval-factory-benchmarking + (`dl/JoC/competitive_evaluation/…`) — re-check before a scored run. Long-context retrieval. Each task is a long multi-turn conversation with N near-identical "needle" responses; the model must reproduce the Nth verbatim @@ -61,6 +64,21 @@ The n3 variants drop over-long samples, so all three are different datasets and candidate, and set it in **both** `data_prep_params` and `collect_rollout_params` — changing one prepares one dataset and rolls out another. +The metric prefix column is the **manifest metric key**: upstream certifies 1M on +`mrcr_n3_1m_benchmark_simple_agent/pass@1/accuracy` and 128K on +`mrcr_n3_128k_benchmark_simple_agent/pass@1/accuracy`. + +**128K is now its own upstream benchmark, on a newer launcher schema.** +`configs/benchmarks/mrcr-128k/` is a separate slug with its own manifest, and its +`bench.yaml` uses the condensed gym block — `benchmark: mrcr/config_n3_128k`, +`runtime_dependencies: tiktoken transformers`, `prepare_args` / `run_args` — against the +`gym_runtime` container, instead of this template's `data_prep_params` / +`collect_rollout_params` / inline `command:` bootstrap. Both reach the same Gym pin +(`a431501a`); the condensed form just moves the bootstrap into the runtime image. This +template stays on the explicit form, which is what the 1M `bench.yaml` still ships. +If you switch a config to the condensed schema, switch the container with it — the +`pre_cmd` and `command:` blocks here have no effect there. + The `num_repeats` column is what each variant **declares upstream**, not necessarily what runs: for `type: benchmark` datasets the value is a placeholder and the runner decides. The template therefore pins `++num_repeats=1` in @@ -90,10 +108,16 @@ deliberately. **Do not change repeat counts when aligning to a golden.** - Fan out via `execution.num_nodes` / `num_instances` (HAProxy pattern A — `references/multi-node.md`). **Size these from the cluster's GPUs-per-node**, do not copy: pick TP for the model, fill the node with DP, then choose instances for - the replica count you want. `parallelism` is the total across instances, so - `--max-num-seqs = ceil(parallelism / num_instances / DP)`. The golden ran 4 nodes - × (TP2 × DP2) on 4-GPU nodes = 8 replicas, `ceil(256/4/2) = 32` each; the same 8 - replicas on 8-GPU nodes is 2 × (TP2 × DP4). + the replica count you want. The golden ran 4 nodes × (TP2 × DP2) on 4-GPU nodes = + 8 replicas; the same 8 replicas on 8-GPU nodes is 2 × (TP2 × DP4). +- **`parallelism` is 512 and is NOT a server cap — do not derive `--max-num-seqs` + from it.** The canonical `bench.yaml` raised it from 256 with the comment that MRCR + prompts run to 1M tokens, "so allow the endpoint to queue requests while keeping + Gym's client concurrency bounded". It sets `++num_samples_in_parallel` and + `++global_aiohttp_connector_limit_per_host`; the server's in-flight cap is + independent and deliberately smaller (the golden's 8 replicas × `--max-num-seqs 32` + = 256, half of 512 — the rest queue). The old `ceil(parallelism / num_instances / DP)` + identity was a coincidence at 256 and is wrong at 512. - **`--max-num-seqs` is a ceiling, not a target.** MRCR is the most KV-bound task in the skill — ~1M input tokens per request against AA-LCR's ~120K — so AA-LCR's rule applies harder: oversubscribe and vLLM preempts, and recomputing a 1M-token @@ -140,28 +164,27 @@ NVIDIA-internal: `modelopttools:eval-config` Step 3d names a working image. ## Canary -MRCR's gym path takes `++limit=N` (the launcher-level `limit_samples` does not -reach the gym). **Not verified on this pinned commit** — treat the first ~30 min -of the real run as the canary. `++limit` caps rollouts -only: the 1M tokenize/drop-over-long **prepare pass still runs in full**, so a -5-sample canary is not cheap. Append it to `collect_rollout_params` in your copy -of the YAML — easier than re-pasting the whole folded scalar through `-o`: - -```yaml - collect_rollout_params: >- - ... - ++limit=5 # canary only — remove for the scored run -``` +**`limit_samples` now reaches the gym.** The canonical `bench.yaml` gates the override +on it — `{% if config.params.limit_samples is not none %}++limit={{config.params.limit_samples}}{% endif %}` +— and the template carries the same line, so `-o ++config.params.limit_samples=5` (or +editing `limit_samples:` in your copy) produces `++limit=5`. -Then watch the first ~30 min of the real run: +`++limit` caps **rollouts only**: the 1M tokenize/drop-over-long **prepare pass still +runs in full**, so a 5-sample canary is not cheap, and `pass@1` from a limited run is +not a score. Also watch the first ~30 min of the real run: ```bash RD=//nemo_gym.0 grep -A1 "=== NeMo Gym commit ===" $RD/logs/client-*.log | grep -c a431501a # pin applied grep -ciE "ModuleNotFoundError|tiktoken" $RD/logs/client-*.log # pre_cmd didn't take wc -l $RD/artifacts/evaluator_rollouts.jsonl # rollouts flowing +ls $RD/artifacts/model_calls | head # observability capture on ``` +`++observability_enabled=true` + `++model_call_capture_dir` are canonical and write +per-call records under `artifacts/model_calls` — useful for diagnosing the prefix-gate +failure below. They grow with the run; drop them only if you are short on space. + **Preempted vs timed out.** A 1M run routinely exceeds 4h. `TIMEOUT` auto-resumes from the response cache; `CANCELLED by ` (preemption) does not — its chained job exits in ~20s (`…finished with 'CANCELLED…' state. EXIT!`), which is expected, @@ -203,8 +226,11 @@ flat. Before quoting, check truncation: `eval_factory_metrics.json` → at `--max-model-len 1100000`). Reference shape (reviewed golden, BF16 Nano 3.5, 1M): `pass@1 = 26.91` (2/4/8 -needles = 36.81 / 27.12 / 16.74), 2363/2363 rollouts, parallelism 256, 4 nodes / -4 instances. Use it to sanity-check shape, not as a bar for another model — a +needles = 36.81 / 27.12 / 16.74), 2363/2363 rollouts, 0 failures, 4 nodes / 4 +instances, Gym `a431501a`, toolchain `0.2.7+20260722.eb3ddf2a` — the sign-off run +recorded in the upstream `manifest.yaml`. That run used `parallelism: 256`; the +canonical `bench.yaml` has since moved to 512, which is a throughput change, not a +scoring one. Use the shape to sanity-check, not as a bar for another model — a rollout count well below 2363 (full runs only — a `++limit` canary is expected to be short) means tasks were lost (e.g. a walltime resume) and the score covers fewer tasks than the reference. diff --git a/plugins/modelopt/skills/evaluation/references/nel-next.md b/plugins/modelopt/skills/evaluation/references/nel-next.md index 8e009098d9f..17a0c7fcf18 100644 --- a/plugins/modelopt/skills/evaluation/references/nel-next.md +++ b/plugins/modelopt/skills/evaluation/references/nel-next.md @@ -52,6 +52,14 @@ prints the resolved spec, so check there if a run behaves unexpectedly. (SWE-bench), and `${MLFLOW_TRACKING_URI}` from `.env`. Run **`modelopttools:eval-config`** (Step 3b) to write them — it holds the canonical values, arch/region rules, and points to the per-benchmark `bench.yaml` source of truth. +- **`HARBOR_ECS_REGION` selects the sandbox region** (`sandbox.region` + + `cluster.container_env.AWS_DEFAULT_REGION`); upstream defaults it to `us-east-1` for + TB2.1 and `us-east-2` for SWE-bench. The two then treat the ECR differently: TB2.1's + repo **name tracks the region** + (`463701203462.dkr.ecr..amazonaws.com/harbor-`), while SWE-bench's is + pinned to `463701203462.dkr.ecr.us-west-2.amazonaws.com/harbor-swebench` whatever the + sandbox region is. So changing the region changes the TB2.1 ECR and not the SWE-bench + one — keep `${HARBOR_*_ECR_REPOSITORY}` in `.env` consistent with it. - `set -a && source .env && set +a` before running so `${VAR}` resolves. ## Architecture — where each piece runs @@ -154,9 +162,13 @@ with its own `run_id`, copying the shared `services:` block. catch. This pair is safe because it is what the TB2.1 + SWE-bench parity runs executed, not by construction. **Re-canary whenever either side moves**, and treat a post-allocation `ValidationError` mentioning an unknown/missing config field as skew, not a config bug. -- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}` → `0.5.0.1-harbor` (multi-arch). Re-check - against `configs/shared/nel_next_containers.yaml` in the eval-factory repo, which is the - pin and does move. Arch-suffixed `0.17.x/0.18.x-harbor-` are too old for TB 2.1. +- **`eval_image`** = `${NEL_NEXT_EVAL_IMAGE}` → `0.5.0.1-harbor` (multi-arch; + `gitlab-master.nvidia.com/dl/joc/competitive_evaluation/nemo-evaluator-next:0.5.0.1-harbor`). + Re-check against `configs/shared/nel_next_containers.yaml` in the eval-factory repo, which + is the single pin for TB2.1 + SWE-bench Verified + SWE-bench Multilingual and does move. + It also carries `nemo_evaluator_next_version`, which the canonical configs echo into the + MLflow tag of the same name — tag your runs with it too, so a score traces back to the + image that produced it. Arch-suffixed `0.17.x/0.18.x-harbor-` are too old for TB 2.1. Private gitlab-master image → cluster needs enroot creds (SKILL Step 7.5). - **`proxy.request_timeout` must be >= `agent_kwargs.llm_kwargs.timeout`** (both 3600). A smaller proxy timeout silently truncates long agent turns. @@ -178,7 +190,16 @@ with its own `run_id`, copying the shared `services:` block. for the life of the run, full request + response bodies, and the whole list is re-serialized on each write. A long agentic run that is 400ing or rate-limiting (the failure this dumps diagnose) grows the proxy without bound — exactly the run you can least afford to lose. - Chain position is per benchmark (last for TB2.1, first for SWE-bench). + It is **first** in the chain for both TB2.1 and SWE-bench (it used to be last for TB2.1). +- **Interceptor lists replace wholesale on merge, they do not append.** A leaf that adds + one interceptor must restate the whole chain from the model fragment + (`http_pairs_dump` → `drop_params` → `consolidate_system` → `reasoning` → + `reasoning_replay`, plus SWE-bench's `system_message` + `turn_counter`). Half a chain is + a silent scoring change, not an error. +- **`proxy.model_traffic.capture_request_body: true` belongs on the service leaf** + (`services..proxy.model_traffic`), never in a shared/benchmark-level block: a + shared `services:` entry is alias-only with no `type`, which fails the service + discriminator in every other model's composed config. - **Mount sources must pre-exist** — pyxis won't create the host side of a bind mount (invisible to `--dry-run`, fails at canary). `ssh 'mkdir -p //.cache/{vllm,huggingface}'`. @@ -188,7 +209,9 @@ with its own `run_id`, copying the shared `services:` block. - **MLflow export — config + a post-run push.** Add `output.export: [mlflow]` + `export_config.mlflow` (hardcode `experiment_name: /` — `${USER}=root` in-container; tags `framework`/`model`/`temperature`/`top_p` + `checkpoint_path`/`benchmark` - for dashboard attribution). `tracking_uri: ${MLFLOW_TRACKING_URI}` (from `eval-config` + for dashboard attribution; canonical also sets `task_name: `, + `nemo-evaluator-next-version`, and `log_config_params: true`). + `tracking_uri: ${MLFLOW_TRACKING_URI}` (from `eval-config` Step 3b — canonical `mlflow.frontier-evals.nvidia.com`; **not** the `mlflow-nemo-evaluator` alias, whose 308 strips `/api/...` → 405). **SLURM does NOT auto-export** — push after the run with `nel-next.sh mlflow-push` (Run flow), which resolves the var and falls back