Meta-Eval (2/2): Report Model Request Usage for Benchmark Runs

[kargibora]

This PR replicates #107 however implements according to OpenRouters official API.

Problem

JudgeArena currently discards token and cost information returned by model providers. Users can see benchmark results, but they cannot tell how many model requests were made or how much reported usage came from generation and judging.

This PR collects provider-reported usage through the shared inference path and attaches one summary to the run metadata. It works for every benchmark using do_inference(), including pairwise, ELO, MT-Bench, fluency, and meta-evaluation.

Before and after

Area	Before	After
Inference result	Returned text and optional top logprobs	Also carries provider-reported request usage
Run summary	No token or cost overview	Reports totals by stage and model
Saved metadata	Usage information was discarded	run-metadata.v1.json includes a usage section
Partial data	Missing provider fields were not visible	Partial token and cost coverage is reported clearly

How it works

benchmark runner
    -> starts one run usage tracker
    -> generation and judging call do_inference()
    -> provider usage is normalized per request
    -> requests are grouped by stage and model
    -> metadata writer saves the current summary
    -> terminal prints the final summary

Generation requests use the generation stage and judge requests use the judging stage. Meta-evaluation only reports judging because it evaluates stored completions and does not generate model answers.

The summary includes request count, input tokens, output tokens, total tokens, reasoning tokens, cached tokens, and cost when the provider returns those fields.

This PR does not estimate missing prices or token counts. If a backend does not report cost, the result says that cost is unavailable. If only some requests contain usage, the totals are marked as partial.

Example

judgearena \
  --task meta-eval-lmarena-100k \
  --judge.model OpenRouter/deepseek/deepseek-v3.2 \
  --meta_eval.top_models 5 \
  --meta_eval.battles_per_model 10

The terminal prints a summary such as:

Model usage:
  Judging: 50 request(s), 120,000 input / 5,000 output tokens, $0.420000
  Total: 50 request(s), 120,000 input / 5,000 output tokens, $0.420000

The same information is stored under usage.total, usage.by_stage, and usage.by_model in run-metadata.v1.json.

Tests

The tests cover provider response parsing, structured inference results, partial cost reporting, metadata output, benchmark-run scoping, and meta-evaluation integration.

Notes

~180 lines are for the tests, thus this PR is minimal and optional (although definitely recommended)

Example

Mergin everything, we can finally report

============================================================
                  🏆 MODEL BATTLE RESULTS 🏆
📊 Task: arena-hard-v2.0-official
🤖 Competitors: Model A: deepseek-r1 vs Model B: gemini-2.0-flash-001+o3-mini-2025-01-31
⚖️ Judge: OpenRouter/openai/gpt-4.1
📈 Results Summary:
   Total Battles: 154 (2×77 — each instruction judged in both orders to detect positional bias)
   Win Rate (A): 57.8%
   ✅ Wins:   86
   ❌ Losses: 62
   🤝 Ties:   6

Per-Category Breakdown:
  Category       | Win Rate(A) | Wins | Losses | Ties
  ---------------+-------------+------+--------+-----
  creative_writing |       93.3% |   56 |      4 |    0
  hard_prompt    |       35.1% |   30 |     58 |    6
📁 Results: parity-results/arena-hard-v2-gpt41-smoke/arena-hard-v2.0-official-deepseek-r1-gemini-2.0-flash-001+o3-mini-2025-01-31-OpenRouter_openai_gpt-4.1-both-20260820_103526
============================================================


Model usage:
  Judging: 100 request(s), 241,932 input / 89,919 output tokens, $1.200336
  Total: 100 request(s), 241,932 input / 89,919 output tokens, $1.200336

Comparison to the official results (https://github.com/lmarena/arena-hard-auto/blob/main/show_result.py)

   Category            JudgeArena    Official, same 50    Difference
  ━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━
   hard_prompt              35.1%                30.6%       +4.5 pp
  ──────────────────  ────────────  ───────────────────  ────────────
   creative_writing         93.3%                95.5%       −2.2 pp