NVIDIA · cjluo-nv · Sep 18, 2026 · coderabbitai · Sep 18, 2026
@@ -42,7 +42,7 @@ for one, do **not** add it to a 0.2.6 `evaluation.tasks` list — instead:
 
 1. Read **`references/nel-next.md`** (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe `recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md`; start from `recipes/examples/example_eval_next.yaml`.
 2. Isolated nel-next venv: `"$SKILL_DIR/scripts/nel-next.sh" --setup-only` (keeps 0.2.6 `nel` untouched).
-3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`) into `.env`; always include the `output.export_config.mlflow` block.
+3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`, `${HARBOR_ECS_REGION}`) into `.env`; always include the `output.export_config.mlflow` block.
 4. Dry-run → canary → full (`nel-next.sh eval run`), then **push to MLflow** — SLURM doesn't auto-export, so run `nel-next.sh mlflow-push -r <run_id> -c <cfg>` after (config-driven; see `references/nel-next.md`).
 
 Steps 1–9 below are currently validated with 0.2.6 — use them for everything else.
@@ -64,6 +64,8 @@ for an "AA" request. If the user asks for MRCR:
 2. **Pick the variant first** (`config_n3_1m` / `config_n3_128k` / `config`) — it
    sets the context cap, dataset *and* metric prefix; the three are not
    comparable; set it in **both** `data_prep_params` and `collect_rollout_params`.
+   Upstream now carries 1M and 128K as two separate benchmarks with their own
+   manifests and metric keys — see the recipe.
 3. `.env`: `HF_TOKEN` (dataset + n3 tokenizer are gated) plus
    `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the `pre_cmd` installs `tiktoken` +
    `transformers`; prepare fails without it) and

@@ -54,6 +54,9 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1
 # AWS_ACCESS_KEY_ID=
 # AWS_SECRET_ACCESS_KEY=
 # NEL_NEXT_EVAL_IMAGE=             # harbor eval image (arch-matched) — set by eval-config
+# HARBOR_ECS_REGION=               # Fargate sandbox region. Upstream defaults: us-east-1
+#                                  # (Terminal-Bench), us-east-2 (SWE-bench). TB's ECR repo
+#                                  # NAME tracks this (harbor-<region>); SWE-bench's does not.
 # HARBOR_ECR_REPOSITORY=           # Terminal-Bench harbor ECR — set by eval-config
-# HARBOR_SWEBENCH_ECR_REPOSITORY=  # SWE-bench harbor ECR (us-west-2) — set by eval-config
+# HARBOR_SWEBENCH_ECR_REPOSITORY=  # SWE-bench harbor ECR (us-west-2, region-independent) — set by eval-config
 # MLFLOW_TRACKING_URI=             # MLflow tracking URI (canonical frontier-evals host) — set by eval-config
@@ -45,17 +45,27 @@ services:
     proxy:
       request_timeout: 3600                      # canonical; MUST be >= benchmarks[].solver.agent_kwargs.llm_kwargs.timeout
       extra_body: {skip_special_tokens: false}   # add model-card sampling extras here if the card specifies them; mirror them in the export tags below
+      model_traffic: {capture_request_body: true}  # FEA-224 — set PER SERVICE, never in a shared block (an alias-only entry has no `type` and fails the service discriminator)
+      # Interceptor lists REPLACE wholesale on merge — if you add one, restate them all.
       interceptors:
+        # FEP-1104/1120 diagnostics — canonical position is FIRST (both TB2.1 and SWE-bench).
+        # Uncomment for a CANARY/debug run, drop it for the scored run: first_n caps only 200s,
+        # so every error pair (full req+res bodies) is retained in memory for the whole run and
+        # re-serialized on each write — unbounded growth exactly when the server errors.
+        # - name: http_pairs_dump
+        #   config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}   # $$ defers expansion to run time
+        # SWE-bench (OpenHands, multi-turn) inserts system_message + turn_counter HERE, before
+        # drop_params — don't lift that chain blind, see swebench_verified.md
         - name: drop_params                        # agents send max_tokens; many servers reject it
           # last two are sent by the 0.5.x harbor eval image; vLLM 400s on them unless stripped
           config: {params: [max_tokens, max_completion_tokens, max_input_tokens_per_task, no_rebuild]}
-        # SWE-bench (OpenHands, multi-turn) adds turn_counter + system_message AND USES A DIFFERENT
-        # ORDER (drop_params before consolidate_system) — don't lift this chain — see swebench_verified.md
-        # FEP-1104/1120 diagnostics — uncomment for a CANARY/debug run, drop it for the scored run:
-        # first_n caps only 200s, so every error pair (full req+res bodies) is retained in memory for
-        # the whole run and re-serialized on each write — unbounded growth exactly when the server errors.
-        # - name: http_pairs_dump                    # canonical LAST in the chain (SWE-bench: first)
-        #   config: {dump_path: "$${NEL_OUTPUT_DIR}/http_pairs_metrics.json", first_n: 50}   # $$ defers expansion to run time
+        - name: consolidate_system
+        # reasoning models only — normalize the reasoning field and replay it across turns.
+        # `reasoning_replay.mode` is per MODEL (think_tags default / native / both), not per
+        # benchmark. Drop both for instruct models.
+        # - name: reasoning
+        # - name: reasoning_replay
+        #   config: {mode: think_tags}
     node_pool: gpu
 
 benchmarks:
@@ -67,10 +77,12 @@ benchmarks:
       timeout_strategy: max          # canonical TB2.1; "task" = leaderboard-comparable
       agent_kwargs: {llm_kwargs: {timeout: 3600}}
     sandbox:
-      region: us-east-1              # MUST match the region in ${HARBOR_ECR_REPOSITORY} (SWE-bench: us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY})
+      # TB2.1's ECR repo NAME tracks the region (harbor-<region>); SWE-bench's is pinned to
+      # us-west-2/harbor-swebench whatever the sandbox region is (us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY}).
+      region: ${HARBOR_ECS_REGION:-us-east-1}
       ecr_repository: ${HARBOR_ECR_REPOSITORY}      # from modelopttools:eval-config
       concurrency: 50
-      log_stream_prefix: terminalbench21-???
+      log_stream_prefix: terminalbench-21-???      # canonical stem + <model>-<framework>
 
 cluster:
   type: slurm
@@ -87,7 +99,7 @@ cluster:
     HF_HOME: /cache/huggingface
     AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
     AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
-    AWS_DEFAULT_REGION: us-east-1     # match sandbox.region / the region in ${HARBOR_ECR_REPOSITORY}
+    AWS_DEFAULT_REGION: ${HARBOR_ECS_REGION:-us-east-1}   # match sandbox.region / the region in ${HARBOR_ECR_REPOSITORY}
     LLM_API_KEY: "no-key-needed"
   mount_home: false
   auto_resume: true
@@ -110,4 +122,12 @@ output:
       exclude_patterns: ["shard*", "model_traffic.jsonl"]   # captured request bodies (FEA-224) stay in the run dir
       description: ???               # '<model> | T=1.0 top_p=0.95 | <benchmark> (timeout_strategy=…) | r8'
       # model/checkpoint_path/benchmark drive dashboard attribution (engine logs only a generic metric key); temperature/top_p mirror generation above.
-      tags: {framework: vllm, model: "???", checkpoint_path: "???", benchmark: "???", temperature: '1.0', top_p: '0.95'}
+      tags:
+        framework: vllm
+        model: "???"
+        checkpoint_path: "???"
+        benchmark: "???"
+        task_name: "terminal-bench-2.1"          # upstream benchmark slug (SWE-bench: swebench-verified)
+        nemo-evaluator-next-version: "0.5.0.1"   # eval_image version, so a score traces to its image
+        temperature: '1.0'
+        top_p: '0.95'
@@ -86,9 +86,11 @@ deployment:
   gpu_memory_utilization: 0.95      # golden raises this from 0.85 to fit the 1M KV cache
   # Golden serves in REASONING mode (thinking forced on via chat_template_kwargs
   # below), so set the model's --reasoning-parser.
-  # After sizing num_nodes / num_instances / TP / DP above, append
-  # `--max-num-seqs N` where N = ceil(parallelism / num_instances / DP) — the GYM
-  # rule, not the generic SKILL one. Golden: ceil(256/4/2) = 32.
+  # After sizing num_nodes / num_instances / TP / DP above, append `--max-num-seqs N`.
+  # Do NOT derive N from `parallelism`: the canonical bench.yaml now sets parallelism
+  # 512 DELIBERATELY ABOVE total server capacity so 1M-token requests queue at the
+  # endpoint instead of leaving replicas idle. Size N from what a replica's KV can hold
+  # (the golden ran 32 per replica × 8 replicas = 256 in flight against parallelism 512).
   # Treat that as a CEILING, not a target. Like AA-LCR (recipes/tasks/aa/lcr.md)
   # this task is KV-bound, and far more so: ~1M input tokens per request vs LCR's
   # ~120K. Too high and vLLM preempts, and recomputing a 1M-token prefill makes the
@@ -131,10 +133,16 @@ evaluation:
       params:
         temperature: 1.0
         top_p: 0.95
-        # Gym client concurrency, TOTAL across instances — not a per-server cap.
-        parallelism: 256
+        # Gym client concurrency, TOTAL across instances — not a per-server cap, and
+        # intentionally ABOVE it: the canonical bench.yaml uses 512 so the endpoint
+        # always has queued work while Gym's own dispatch stays bounded. Drives both
+        # ++num_samples_in_parallel and ++global_aiohttp_connector_limit_per_host.
+        parallelism: 512
         request_timeout: 36000      # 1M-token prefills are slow
         max_retries: 10
+        # Empty (null) = full run. Set N for a canary: MRCR forwards it to the gym
+        # as ++limit (see the gated line in collect_rollout_params below).
+        limit_samples:
         # Uncapped: answers reproduce a full earlier turn, and any cap truncates
         # it and craters SequenceMatcher.ratio (see max_output_tokens=null below).
         max_new_tokens:
@@ -292,9 +300,12 @@ evaluation:
                   ++reuse_existing_data_preparation=true
                   ++skip_venv_if_present=true
                   ++upload_rollouts_to_wandb=false
+                  ++observability_enabled=true
+                  ++model_call_capture_dir={{config.output_dir}}/model_calls
                   ++responses_create_params.temperature={{config.params.temperature}}
                   ++responses_create_params.top_p={{config.params.top_p}}
                   ++responses_create_params.max_output_tokens=null
+                  {% if config.params.limit_samples is not none %}++limit={{config.params.limit_samples}}{% endif %}
                 common_params: >-
                   ++num_repeats=1
                   ++policy_base_url={{target.api_endpoint.url}}

@@ -8,6 +8,11 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`.
 > **Source of truth:** `configs/benchmarks/swe-bench-verified/bench.yaml` in
 > nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the eval-image
 > pin in `configs/shared/nel_next_containers.yaml` — match its values for a reference run.
+>
+> That dir also ships `bench_direct.yaml`, a **different backend**: the Gym/opencode
+> route (`++num_samples_in_parallel=1024`, its own Gym commit pin, opensandbox) used by
+> the P/D recipes. `benchmark_backend: evaluator` — the harbor path described here — is
+> what this skill runs. Don't mix values between the two.
 
 ## Task-specific values (canonical `bench.yaml`)
 
@@ -16,12 +21,15 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`.
 | `playbook` | `swebench_verified` (`harbor://swebench-verified@1.0`) |
 | agent | `openhands-sdk` (playbook; `agent_kwargs: {max_iterations: 200, version: "1.17.0"}`) |
 | scope | 500 Python tasks × `repeats: 5` |
-| `max_concurrent` / `sandbox.concurrency` | `15` in `bench.yaml`; per-model configs override it (MiniMax-M2.7 uses `20`) |
+| `max_concurrent` / `sandbox.concurrency` | `15` (canonical `bench.yaml`); the reviewed per-model leaves keep it. A separate customer-launch track (`reviewed_configs/clu/…`) runs `repeats: 3` / `concurrency: 10` — **not** AA-comparable, don't borrow its numbers |
 | `solver` | `timeout_strategy: max`, `run_timeout: 10800` (3h), `agent_kwargs.llm_kwargs.timeout: 3600` |
-| `sandbox.region` | `us-east-2` |
-| `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` (dedicated `harbor-swebench` repo, **us-west-2**, regardless of sandbox region) |
+| `sandbox.region` | `${HARBOR_ECS_REGION:-us-east-2}` (differs from TB2.1's `us-east-1` default) |
+| `sandbox.ecr_repository` | `${HARBOR_SWEBENCH_ECR_REPOSITORY}` → `463701203462.dkr.ecr.us-west-2.amazonaws.com/harbor-swebench`. **Hardcoded us-west-2 upstream** — unlike TB2.1 the repo does *not* track `HARBOR_ECS_REGION`; moving the sandbox region leaves the ECR alone |
 | `cluster.eval_image` | `${NEL_NEXT_EVAL_IMAGE}` → **`0.5.0.1-harbor`** (same pin as TB2.1: `configs/shared/nel_next_containers.yaml`) *(shared — see `references/nel-next.md`)* |
-| `cluster.container_env.AWS_DEFAULT_REGION` | `us-east-2` (match `sandbox.region`) |
+| `cluster.container_env.AWS_DEFAULT_REGION` | `${HARBOR_ECS_REGION:-us-east-2}` (match `sandbox.region`) |
+| `sandbox.log_stream_prefix` | canonical stem `swebench-<model>-<cluster>-<framework>` |
+| `output.export_config.mlflow.tags` | canonical adds `task_name: swebench-verified` and `nemo-evaluator-next-version: 0.5.0.1` |
+| `proxy.model_traffic.capture_request_body` | `true`, set **per service** (`services.<alias>.proxy.model_traffic`) — never in a shared block |
 | `instruction_template` | `/configs/prompts/swebench_instruction.md`, **must be MOUNTED**; content is scoring-relevant (gotcha below) |
 | `proxy.request_timeout` | `3600` (FEP-1104 paired HTTP timeout; leaves mirror it on the service proxy) *(shared — see `references/nel-next.md`)* |
 | `drop_params` | `max_tokens`, `max_completion_tokens`, `max_input_tokens_per_task`, `no_rebuild` *(shared — see `references/nel-next.md`)* |
@@ -32,15 +40,15 @@ ECR/region. Start from `recipes/examples/example_eval_next.yaml`.
 benchmarks:
   - playbook: swebench_verified
     repeats: 5
-    max_concurrent: 15            # keep == sandbox.concurrency; per-model configs may raise both
+    max_concurrent: 15            # canonical; keep == sandbox.concurrency
     instruction_template: /configs/prompts/swebench_instruction.md   # mounted (see gotcha)
     solver:
       service: <svc-name>
       timeout_strategy: max          # canonical; "task" = leaderboard-comparable
       run_timeout: 10800
       agent_kwargs: {llm_kwargs: {timeout: 3600}}
     sandbox:
-      region: us-east-2
+      region: ${HARBOR_ECS_REGION:-us-east-2}   # canonical; the ECR below stays us-west-2
       ecr_repository: ${HARBOR_SWEBENCH_ECR_REPOSITORY}
       concurrency: 15
       log_stream_prefix: swebench-verified-<model>-<cluster>
@@ -88,10 +96,16 @@ OpenHands runs ~200 turns/task. The canonical config adds a `system_message`
 interceptor (a large OpenHands system prompt — copy it verbatim from `bench.yaml`)
 plus `turn_counter`.
 
-**Order differs from TB2.1**: `http_pairs_dump` is **first** (not last) and `drop_params`
-comes **before** `consolidate_system`. `http_pairs_dump` is canary/diagnostic-only — it
-retains every error pair in memory for the whole run (`references/nel-next.md`); drop it
-from the scored config.
+**Order.** `http_pairs_dump` is **first**, then `system_message` → `turn_counter` →
+`drop_params` → `consolidate_system` → `reasoning` → `reasoning_replay`. (TB2.1 now puts
+`http_pairs_dump` first too; the SWE-bench-only part is the `system_message` +
+`turn_counter` pair.) `http_pairs_dump` is canary/diagnostic-only — it retains every error
+pair in memory for the whole run (`references/nel-next.md`); drop it from the scored config.
+
+**Interceptor lists replace wholesale on merge.** The reviewed leaf restates the model
+fragment's whole chain to add two entries — it does not append. Adding `system_message`
+without re-listing `drop_params`/`consolidate_system`/`reasoning`/`reasoning_replay`
+silently drops them.
 
 ```yaml
 proxy:
@@ -127,11 +141,21 @@ Fargate quota and `N × gpus_per_node` against your allocation.
 
 ## Score Extraction
 
-Report **`pass@1`** only — benchmark `swebench-verified@1.0`, scorer `pass@1` (0–1):
+Report **`pass@1`** — benchmark `swebench-verified@1.0`, scorer `pass@1` (0–1):
 the resolved rate over the 500 tasks, **already averaged over repeats** (nel-next
 reports a single `pass@1`; there is **no `avg-of-N` key** like the 0.2.6 nemo-skills
-metrics). MLflow logs it as `pass_at_1`. Read from `report.md` (Benchmark / Scorer
-table) in the run dir or `nel eval report -r <run_id>`, then push to MLflow with
+metrics). MLflow logs it as `pass_at_1`.
+
+> The upstream `manifest.yaml` lists its metric key as **`mean/reward`**, not `pass@1` —
+> that is the key the Gym/direct (`bench_direct.yaml`) recipes emit, and it is shared by
+> the whole SWE-bench family (Verified / Multilingual / Pro). On the harbor path the
+> certificates in `reviewed_configs/…/swebench_verified/certificates/` record `pass@1`
+> (e.g. `0.516`), so `pass@1` is still what you report for a nel-next run. Quote the key
+> name alongside the number so a `mean/reward` from the direct path is never compared
+> against it.
+
+Read from `report.md` (Benchmark / Scorer table) in the run dir or
+`nel eval report -r <run_id>`, then push to MLflow with
 `nel-next.sh mlflow-push -r <run_id> -c <cfg>` (SLURM doesn't auto-export). Keep
 `timeout_strategy` + the instruction/system prompt fixed across baseline vs quantized
 for a valid delta.