Document the LLM Judge Scorer surface, and add the CI that keeps it true

[michalakmarcinth]

0.6.36 consolidated grading configs into a single entity — a judge rubric plus an offline profile and an online profile, at client.monitor.judge_scorers — and soft-deprecated the two half-views that used to reach it.

None of that reached the docs. EVALUATIONS.md, TRACING.md, README.md and CICD_EVAL.md had no mention of a judge scorer between them, so the only way to find the surface was to read monitor/judge_scorers.py. Two commits: document it, then add the job that would have caught the omission.

1. Document it

EVALUATIONS.md gains the section, replacing the one that documented the legacy settings builder as if it were the primary surface: the three sections and what each holds, the CRUD calls and what sparse update() does with online, giving a scorer an online profile with live=True, and the calibration/tuning/history calls that resolve the online profile from the scorer id.

TRACING.md marks online_evaluators as the online half-view and links across. README.md gets a short pointer next to the online-evaluator example so the front door names the unified entity. Both legacy sections stay — they document surfaces that still work, and on engines without the new route they are the only ones that do.

Two things stated that aren't in any docstring

Which grader kwarg to write. scorer_id is the current name and evaluation_settings_id is the pre-consolidation spelling of the same id — but they are not interchangeable across versions:

Kwarg	agentx-python < 0.6.36	>= 0.6.36
evaluation_settings_id	works	works
scorer_id	TypeError	works

So new code should say scorer_id, and anything that may run under a pinned older client — a CI gate, a committed evaluation harness re-run for a controlled before-and-after — should keep the older spelling.

Engine compatibility. client.monitor.judge_scorers calls /agent-monitoring/judge-scorers, and a self-host engine that doesn't serve that route returns 404 while the rest of /agent-monitoring/ answers normally. Verified against a running engine:

Endpoint	Result
/agent-monitoring/online-evaluators	200
/agent-monitoring/patterns	200
/agent-monitoring/judge-scorers	404
/custom-agent-evaluations/evaluation-settings	200, 10 seeded configs

client.monitor.judge_scorers.list() fails identically there. The section says so, shows the check, and points at the legacy views as the portable path — so nobody builds on the surface and finds out in production.

2. Add the CI that keeps it true

The surface shipped undocumented because nothing in this repo checks the docs — release.yml only publishes.

tests/test_docs_match_sdk.py extracts the fenced python from the four docs and resolves what they show against the installed package: every judge_scorers method, every builder keyword, every keyword on evaluations.run, that the legacy views the docs promise still work are still there, that both grader spellings resolve to one id and two different ids raise, and that cross-document links point at headings that exist. Prose is out of scope on purpose — only fenced blocks are read, because that's what a reader copies.

Each sweep asserts it still found roughly what it found when written. A regex that quietly stops matching is the failure mode that matters here: it doesn't go red, it goes green while checking nothing.

Proven to fire, not assumed. Against a copy of the tree — a documented method the SDK dropped, a builder keyword that no longer exists, a run keyword that no longer exists, and a heading renamed out from under its links. Each fails exactly one test; the restored tree passes.

docs.yml runs it on PRs and pushes to main, on 3.9 (what release.yml builds the wheel with) and 3.12.

On the scope of that job

It runs this file plus the judge-scorer unit tests, not the whole suite. tests/test_integrations.py and tests/test_span_tree.py need optional extras (google-adk) and fail on ImportError without them — four failures on a clean checkout today. A badge that's red for reasons unrelated to the docs would say nothing about the docs. Widening it wants those extras installed first, and is its own change. The reasoning is commented in the workflow rather than left implicit.

Verification

• Docs claims resolved against 0.6.36 from PyPI: 149 call sites, keywords and claims, including all eleven judge_scorers methods and every builder keyword
• New test suite: 21 passed, run with the exact CI command in a fresh virtualenv
• All four staged breaks caught, each failing exactly one test
• Test file parses under 3.9 and 3.12; docs.yml validated as YAML

One claim was softened during review rather than shipped: an earlier draft asserted that client.evaluations.settings.get(scorer.id) returns the offline half of a scorer created through the unified surface. That round-trip can't be observed on an engine that 404s the unified route, so the text now states only what the SDK's own design guarantees — that the ids address the same records.

Docs and tests only; no version bump.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WYi6akpDa8qfiojRed2cAE