Public benchmark surface: one doc, one results table, honest claims

[deepfates]

Every number this repository publishes now lives in one table with the command that produces it, and every claim that cannot be re-measured by an outsider says so in its own first paragraph.

Supersedes #128, which GitHub closed when #126 merged and its base branch was deleted. Same two commits, replayed onto main at 547013b.

What a stranger with an API key can now do

benchmarks/RESULTS.md has one row per number: value, confidence interval, dataset and its license, model, provider, date, commit, command. Prose docs cite rows instead of restating numbers. The table is split three ways — re-measurable, recomputable only, and findings that are not results — because those are different things and blurring them is how benchmark claims go bad.

docs/BENCHMARKS.md is the index: what each command needs (key, Python environment, time, rough cost), what it measures, and what it deliberately does not. It ends with a section called "Cannot be re-measured" that names the GEPA six-family campaign, the stopped IFBench rehearsal, and the historical negatives, and says plainly why each is unrunnable here. Every mix command it publishes is checked by a test to be a real task or alias.

Things that turned out not to be true

The GEPA six-family campaign had no split files at all — not three of six, none. families.json pins six families by per-split SHA-256 and the data those digests describe is absent. That is now stated in the attribution note and in the "cannot be re-measured" section.

ifbench_instruction_following is not IFBench. Its rows are written in bench/imp/benchmark_truth/fetcher.ex under the internal name imp/local-ifbench, config verifier-smoke. A score on them is a score on our own verifier rows. benchmarks/data/IFBENCH_ATTRIBUTION.md corrects this and separates it from the matched examples, which do use real IFBench with an unknown license.

docs/EVIDENCE.md claimed DSPy 3.3.1 in one place and 3.2.1 in another. The harness uses 3.2.1 at 29448ae12756abdd14bd8796c819247ebb83673c; two newer probes use 3.3.1 and now say so where they appear. The agent-optimization cost was given to six decimals with no model named; it was openrouter:openai/gpt-5.4-mini for task calls and anthropic/claude-sonnet-4.6 for reflection, on 2026-08-23 at 8a6ce8fd.

The differential CI job carried a comment about known-red residue. Re-run on a clean tracked tree, neither residue is red. The comment is replaced with what was measured and the 2026-08-07 row in decisions.md is retired with today's date.

The TREC recomputation

recompute_compact.exs used to print GEPA +0.4000, MIPROv2 +0.1458, GEPA Imp-minus-DSPy -0.0083 as string literals and assert equality. It now derives those three quantities, prints what it computed, and exits non-zero with both sides printed when they disagree. A new test tampers with the committed aggregate and asserts the command fails — so the command cannot silently become a stored answer again.

#126 deletes the scored rows the command reads. They are restored under examples/matched_instruction_optimizers_trec/data/, beside the aggregator that consumes them, with identical digests. test/case_study_trec_recomputation_test.exs is restored too: it is not a digest check, it shells out and runs the aggregation end to end.

docs/CASE_STUDY_TREC.md now opens by saying the result is recomputable but not reproducible, and why: the raw provider traces are 181 MB and were not published, so the step from "a model answered" to "a row records this answer" is taken on our word.

Dataset licenses

Notes modelled on HOTPOTQA_ATTRIBUTION.md for the sets that had none. Where the license cannot be determined it says unknown and points at what was checked: TREC's pinned dataset card declares none, the GEPA artifact repository declares none for three of six families, and IFBench's corpus has none either. The tutorial tickets were written here and are MIT. The deployment example's data already declared CC-BY-4.0 and CC-BY-SA-4.0 inside its receipts; the new README surfaces that.

docs/internal

Renamed to docs/differentials/, because "internal" tells a public reader the directory is not for them when most of it documents commands they can run. Sixteen files stay, each naming a differential that still exists, cleaned of references to paths #126 deletes and of vocabulary that named bookkeeping rather than the pinned upstream. Eight are gone: the coverage matrix and upstream conformance audit (release truth tables), the 2026-08-07 trust audit (stale, and its "hard NO-GO" must not go public), the identity cutover note (no command), and the three documents whose runnable content is folded into docs/BENCHMARKS.md. The optimizer trajectory contract had no runnable command but was public API prose rather than bookkeeping, so it moved into docs/TRAJECTORIES.md instead of being deleted.

Verified

• mix format --check-formatted — clean
• mix test test/documentation_contract_test.exs test/learning_path_contract_test.exs test/tutorial_parity_test.exs test/case_study_trec_recomputation_test.exs test/package_contract_test.exs — 50 tests, 0 failures
• MIX_ENV=test mix docs.check — docs render, 5/5 livebooks pass
• The TREC recompute command from docs/CASE_STUDY_TREC.md — printed the documented values and agreed with the committed aggregate; tampering with the aggregate makes it exit 1

The tutorial experiment was not run. It has no provider-free mode, and no OPENAI_API_KEY or OPENROUTER_API_KEY is present in the environment or in an ignored .env. Rows R1 and R2 ship from the committed run of 2026-08-22 at 88d61a9c, labelled with that date.