feat(config): consolidate all time knobs into settings.timeouts; make --timeout a real run watchdog

[viraatc]

What does this PR do?

Unifies every global time knob into one place — the frozen Timeouts model in the new src/inference_endpoint/config/timeouts.py, mounted at settings.timeouts — gives the previously dead --timeout flag real whole-run-watchdog semantics, deletes the --duration knob, and splits the ~1700-line config/schema.py into focused domain modules.

The headline bug this fixes

BenchmarkConfig.timeout (--timeout, top-level timeout: YAML key) was consumed nowhere — a silent no-op. It is now settings.timeouts.run_timeout_s: a real whole-run watchdog whose deadline is captured at run_benchmark entry (setup counts against it) and stays armed through every phase and the metrics drain. When it fires: the session stops (ENDED still flows, buffered tokenizer-drain samples are recorded), the metrics aggregator is SIGTERMed (its handler writes an INTERRUPTED final_snapshot.json), scoring is skipped, and run_benchmark raises ExecutionError after artifacts are written — a fired watchdog can never yield complete: true and always exits non-zero. Locked by tests/integration/commands/test_run_timeout.py.

The one block

settings:
  runtime:
    max_duration_ms: null            # workload cap; caps perf-phase ISSUING, normal end, valid report
  timeouts:                          # every global wait/deadline
    run_timeout_s: null              # --timeout; whole-run watchdog -> INTERRUPTED + non-zero exit
    service_ready_timeout_s: 30.0    # metrics/event-logger startup
    warmup_drain_timeout_s: 240.0    # per-phase post-issuing drains (null = wait indefinitely)
    performance_drain_timeout_s: null
    accuracy_drain_timeout_s: null
    metrics_drain_timeout_s: null    # tokenization drain; EXPIRY FAILS THE RUN (complete: false + non-zero exit)
  client:
    worker_initialization_timeout: 60.0   # endpoint-client internals stay on client (unchanged from main)
  metrics_tokenizer_workers: 2       # non-time knob; flat Settings field

Deleted outright (hard cutover, extra=forbid makes stale YAML keys error loudly): top-level timeout:, the settings.drain block (fields absorbed into timeouts with consistent *_drain_timeout_s names), settings.service_ready_timeout_s (moved), runtime.min_duration_ms/--duration, and every 0 = unlimited sentinel (gt=0; unlimited is null).

docs/CLI_QUICK_REFERENCE.md gains a run-lifetime timeline showing where every knob acts, a YAML-path <-> CLI-flag table, and composition rules. All CLI aliases unchanged.

Sample count: explicit or dataset-once (breaking)

--duration/min_duration_ms is deleted (review consensus in the duration thread): the sample count is explicit runtime.n_samples_to_issue (--num-samples) or, when omitted, one pass over the dataset.

⚠️ Bare-config default changed: previously a config with neither knob derived target_qps x 10 min worth of samples; it now runs the dataset once. Example YAMLs that relied on derivation carry explicit counts. The MLPerf ruleset path (RuntimeSettings/UserConfig) keeps its internal duration fields, so submission validity rules are unaffected.

Metrics drain gets teeth

An expired metrics_drain_timeout_s (aggregator finalizes COMPLETE with n_pending_tasks > 0) previously exited 0 with complete: false buried in the summary. It now fails the run: artifacts are written first, then ExecutionError. Partial ISL/OSL/TPOT stats can never look like a clean run. (The audit path already refused to certify these.)

schema.py split

config/schema.py now owns only the root aggregates (BenchmarkConfig/EndpointConfig), cross-field validation, the root-level TestType/TestMode, and the re-export hub (import sites unchanged). Domains moved to settings.py, timeouts.py, model_params.py, datasets.py, audit.py — each module owns its models AND enums (no shared enums file; every enum had exactly one consumer). Dead code deleted: SystemDefaults, TEMPLATE_TYPE_MAP.

Design invariants

• run_timeout_s never derives per-stage deadlines; it is the only total-wall-time bound.
• max_duration_ms and performance_drain_timeout_s never run concurrently: the cap bounds issuing and skips the drain; the drain bounds post-issuing waiting after a natural end.
• Worker lifecycle timeouts are endpoint-client internals and stay on settings.client (unchanged from main). turn_timeout_s stays dataset-scoped.
• Hot path untouched: everything resolves to plain values at setup.
• No back-compat shims: old keys hard-error; templates/examples/docs/tests migrated in this PR.

Robustness fixes found during review hardening

• publish_final is serialized with an asyncio.Lock: a watchdog SIGTERM racing the ENDED-driven finalize can no longer abandon a half-written final_snapshot.json.
• A watchdog firing before the session exists still publishes STARTED/ENDED (the stopped session runs with zero samples), so the event logger exits and the drain cannot hang.
• finalize_benchmark forces complete: false when the aggregator finalized COMPLETE just before the SIGTERM landed — timed-out artifacts are never split-brained.

Follow-up filed: #449 (promote warmup to a first-class phase type with per-phase config).

Type of change

• Bug fix
• New feature
• Refactor/cleanup

Testing

• Tests added/updated — watchdog integration tests (mid-run abort, normal completion, abort during metrics drain, drain-timeout failure), dataset-once default, 0-sentinel rejection, removed-key rejection, launcher targeted-terminate; all suites migrated
• All tests pass locally — tests/unit: 1865 passed; integration command suites green
• Manual testing completed — every example YAML validates against the new schema; CLI --help verified aliases unchanged

Checklist

• Code follows project style
• Pre-commit hooks pass
• Documentation updated — AGENTS.md, CLI docs (+ time-knob timeline), LOCAL_TESTING, config DESIGN docs, compliance audit plan, example READMEs

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request refactors the configuration schema by centralizing all global durations, deadlines, and timeouts into a new frozen Pydantic model Timeouts (accessible via settings.timeouts). This separates workload durations from failure-handling deadlines. Additionally, a whole-run watchdog (run_timeout_s) has been introduced to gracefully abort stuck runs, signaling managed subprocesses via SIGTERM to write an interrupted final snapshot before exiting non-zero. All configuration templates, examples, and tests have been updated to align with this new schema, and new integration tests have been added to verify the watchdog behavior. No review comments were provided, so there is no feedback to address.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[nvzhihanj]

Do you mean if both are set, both will be omitted?
What is the current loadgen behavior? I am thinking whether taking the max of the 2 will make more sense or we should error out

[arekay-nv]

Can we drop duration - this is overloading with the timeout as the duration of a benchmark can be the bounded max-runtime. We can rely on num-samples to specify exactly how many samples the user wants to run.
This will bound the lower end - and we can use the timeout on the upper end.

I think this would make it easier to understand rather than having to recall the details every time we specify duration, timeout and num-samples.

[roborluo]

I think he is trying to say if we set neither, it will just sweep once for the dataset we sepcified ? But yeah, a little unclear at initial read.

Also +1 to @arekay-nv . Having --min-duration-ms --duration representing the same it confusing

[viraatc]

Do you mean if both are set, both will be omitted?

this originally was is both are set, raise validation error - nothing is run.
if neither is set, we would run 1 dataset epoch.
if any is set, that value would be used

Can we drop duration

+1, dropped: duration and min_duration_ms so no longer applies.

[arekay-nv]

I think the schema breakdown make sense and is a lot cleaner.
Regarding the timeouts - two suggestions, and feedback is welcome:

1. Remove the duration field - makes it simpler especially since we are mostly going to be doing concurrency based runs.
2. Modularize the phases with an explicit type of phases and dependencies, but move the per-phase timeouts/drains etc there.
   So a global timeout for everything - and a per-phase config for controlling how a phase behaves. We can have some explicit dependencies such as warmup always goes before performance, reporting comes after accuracy etc.

[arekay-nv]

Can we drop duration - this is overloading with the timeout as the duration of a benchmark can be the bounded max-runtime. We can rely on num-samples to specify exactly how many samples the user wants to run.
This will bound the lower end - and we can use the timeout on the upper end.

I think this would make it easier to understand rather than having to recall the details every time we specify duration, timeout and num-samples.

[roborluo]

wonder why you seperate part of the config and left some in the schema.py? What is your critira to do the splitting?

[viraatc]

The intent was to shrink schema.py (~1700 lines and growing) and refocus it on what it's actually for: the top-level aggregates — BenchmarkConfig and EndpointConfig — plus the cross-field validation that has to see every domain at once. Everything else moved out along one rule: one module per config domain, every name (model, enum, helper) beside its owner.

 ▏ ┌─────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
 ▏ │ file            │ contains                                                                                                                                                                 │
 ▏ ├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 ▏ │ schema.py       │ BenchmarkConfig/EndpointConfig (root aggregates), cross-field validation, root-level TestType/TestMode, and the re-export hub — existing config.schema imports unchanged │
 ▏ ├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 ▏ │ settings.py     │ the settings: block: Settings, RuntimeConfig, LoadPattern(+LoadPatternType), WarmupConfig, ProfilingConfig(+ProfilerEngine), EarlyStoppingConfig                         │
 ▏ ├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 ▏ │ timeouts.py     │ Timeouts — every global wait/deadline (settings.timeouts)                                                                                                                │
 ▏ ├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 ▏ │ model_params.py │ ModelParams(+StreamingMode), OSLDistribution(+OSLDistributionType), SubmissionReference                                                                                  │
 ▏ ├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 ▏ │ datasets.py     │ Dataset(+DatasetType), AccuracyConfig(+EvalMethod/ScorerMethod), AgenticInferenceConfig, the generation-override merge helpers                                           │
 ▏ ├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 ▏ │ audit.py        │ AuditTestId, OutputCachingTestConfig (the audit: block)                                                                                                                  │
 ▏ └─────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

[viraatc]

Addressed the granularity concern directly — the split is now three modules, not six. Criterion: keep together what changes together. settings.py owns the settings: subtree (including the frozen Timeouts at settings.timeouts); workload.py owns the request payload + data definition (model_params:, datasets:, accuracy config, and the generation-override merge helpers whose only consumer is Dataset); schema.py keeps the root aggregates, the cross-field validation that must see every domain at once, the small root-level audit: block, and the re-export hub — so no import site outside config/ changed. Also fixed while consolidating: the regenerate-templates pre-commit hook now matches every config/*.py instead of an enumerated filename list, so a renamed module can never silently skip template regeneration.

[roborluo]

can we add a normal completion before timeout? And another timeout during metric darining?

[viraatc]

Both cases are covered in tests/integration/commands/test_run_timeout.py: test_generous_run_timeout_completes_normally (normal completion with the watchdog armed) and test_run_timeout_during_metrics_drain_interrupts (watchdog firing during the metrics drain), plus test_metrics_drain_timeout_fails_run for the drain-deadline-expiry path.