Public benchmark surface: one doc, one results table, honest claims

.dialyzer_ignore.exs

@@ -37,7 +37,7 @@
    {286, 30}},
   # defensive clause for non-covered differential rows
   {"bench/imp/benchmark_truth/optimize_anything/upstream_differential.ex", :pattern_match_cov,
-   {228, 10}},
+   {227, 10}},
   # MapSet opacity on case-id sets typed through campaign JSON (col 21)
   {"bench/imp/benchmark_truth/rlm_campaign.ex", :call_without_opaque, {813, 21}},
   # MapSet opacity on case-id sets typed through campaign JSON (col 53)
@@ -108,8 +108,6 @@
   {"lib/imp/tracking/mlflow.ex", :call_without_opaque, {361, 8}},
   # defensive clause dialyzer pins to the module head (line 1)
   {"lib/mix/tasks/imp.benchmark.fast_slow.ex", :pattern_match, 1},
-  # MapSet opacity on the hop-id set from analysis JSON
-  {"lib/mix/tasks/imp.benchmark.hotpotqa_analysis.ex", :call_without_opaque, {159, 24}},
   # MapSet opacity on the run-index set from parity JSON
   {"lib/mix/tasks/imp.benchmark.parity.aggregate.ex", :call_without_opaque, {328, 24}},
   # raise-only helper: invalid_python!/1 exists to Mix.raise

.github/workflows/ci.yml

@@ -197,7 +197,7 @@ jobs:
       # Its self-test imports gepa_artifact from the checkout root; the
       # maintainer environment had that on the ambient path, runners don't.
       - run: PYTHONPATH="$PWD/tmp/gepa-artifact" scripts/setup_corrected_gepa_comparator.sh tmp/gepa-artifact
-      # Source-exact IFBench scorer venv (docs/internal/BENCHMARK_TRUTH.md).
+      # Source-exact IFBench scorer venv (docs/BENCHMARKS.md).
       - run: |
           python3 -m venv tmp/ifbench-parity-venv
           tmp/ifbench-parity-venv/bin/python -m pip install -q -r benchmarks/requirements-ifbench-parity.txt
@@ -208,10 +208,11 @@ jobs:
       # spend their test timeout compiling.
       # Children run in both dev (banking77) and test (hotpotqa) envs.
       - run: cd examples/deployment && IMP_PATH=../.. mix deps.get && IMP_PATH=../.. mix deps.compile && IMP_PATH=../.. mix compile && IMP_PATH=../.. MIX_ENV=test mix deps.compile && IMP_PATH=../.. MIX_ENV=test mix compile
-      # Known-red residue at introduction (tracked on imp-sqkr): the sealed
-      # v1 contract detects mix.lock drift after the bandit CVE bump, and the
-      # MuSiQue receipts read a machine-local /tmp data dir. This job makes
-      # that visible on every run rather than hiding it in the fast lane.
+      # Re-measured 2026-09-17 on a clean tracked tree: neither residue this
+      # comment used to describe is still red. The mix.lock drift is gone and
+      # the MuSiQue receipts are excluded by their own tag. The only remaining
+      # failures came from Imp.UpstreamFidelityTest, which is removed with the
+      # rest of the conformance-ledger machinery.
       - run: mix differential.check
 
   docs:

.gitignore

@@ -12,16 +12,13 @@
 
 # Replaceable build and test material.
 /tmp/
-/benchmarks/data/
-!/benchmarks/data/
-/benchmarks/data/*
-!/benchmarks/data/musique-ans-mipro-current-v1.receipt.json
+# `mix imp.benchmark.fetch` writes split-sized dataset files here. Everything
+# else under benchmarks/data -- fixtures, manifests, provenance, build
+# scripts -- is tracked, so nothing a checkout needs is ignored.
+/benchmarks/data/*-[0-9]*.jsonl
 !/benchmarks/data/hotpotqa-validation-0-10.jsonl
-!/benchmarks/data/hotpotqa-validation-0-10.manifest.json
-!/benchmarks/data/HOTPOTQA_ATTRIBUTION.md
 
-# Disposable executions and resumable campaign state. Immutable accepted
-# evidence lives outside these roots under benchmarks/evidence/admitted.
+# Benchmark output. Every task writes its report under one of these roots.
 /benchmarks/runs/
 /benchmarks/checkpoints/
 /benchmarks/results/

CONTRIBUTING.md

@@ -26,19 +26,17 @@ run under `mix test test/learning_path_contract_test.exs`, and
 both green when changing public examples or notebooks.
 
 Provider-backed and research-scale tests are separate because they require
-credentials, external services, canonical datasets, or significant spend. See
-`docs/maintainers/RELEASE.md` and `docs/maintainers/EVIDENCE.md` before changing
-a provider, optimizer, benchmark, or fidelity claim. `.env.example` documents
+credentials, external services, canonical datasets, or significant spend.
+`.env.example` documents
 the supported live-test variables; keep real credentials in an ignored `.env`.
 
 ## Maintainer checks
 
-The benchmark-evidence and reproduction-registry tests are excluded from the
-default `mix test` run (tag `:evidence_infrastructure`). They validate the
-committed benchmark artifacts against full git history, the pinned DSPy Python
-environments (`scripts/setup_dspy_parity_env.sh` and friends), and in some
-lanes a `.env` with provider credentials — none of which a fresh clone has.
-To run them:
+Some benchmark and upstream-differential tests are excluded from the default
+`mix test` run (tag `:evidence_infrastructure`). They need the pinned DSPy
+Python environments (`scripts/setup_dspy_parity_env.sh` and friends), and in
+some lanes a `.env` with provider credentials — neither of which a fresh clone
+has. To run them:
 
 ```sh
 scripts/setup_dspy_parity_env.sh
@@ -47,20 +45,14 @@ scripts/setup_reference_test_env.sh
 EVIDENCE_INFRASTRUCTURE=1 mix test        # or: mix test --include evidence_infrastructure
 ```
 
-They also need a full (non-shallow) clone, because the source-binding
-validators resolve ancestor commit SHAs.
-
 ## Maintainer Authority
 
 Public behavior belongs to code, tests, and user documentation. Pinned upstream
-semantics belong to `benchmarks/authorities.json`; reproducible research
-protocols and retained results belong to `benchmarks/reproductions.json` and
-`benchmarks/evidence/`. Unfinished work is a pull request on a topic branch; there is no ticket file in this repository.
-
-`benchmarks/claims.json` is a scoped index for auditing unusually broad or
-comparative statements. It is not a release score. Read
-`docs/maintainers/AUTHORITIES.md`, `docs/maintainers/REPRODUCTIONS.md`, and
-`docs/maintainers/RESEARCH_PROTOCOLS.md` when changing those surfaces.
+semantics belong to `benchmarks/authorities.json`: the differential harness
+reads it before comparing Imp against upstream, and it fails closed when a pin
+drifts. Benchmark results belong to whoever ran the harness, in the report the
+task writes. Unfinished work is a pull request on a topic branch; there is no
+ticket file in this repository.
 
 ## Design Standard
 
@@ -70,6 +62,9 @@ comparative statements. It is not a release score. Read
 - Preserve upstream algorithmic semantics when using an upstream name.
 - Give deliberate Elixir-native alternatives a distinct contract and rationale.
 - Never turn fixtures, smoke tests, or symbol presence into broad parity claims.
+- Every published number needs a row in `benchmarks/RESULTS.md` carrying its
+  dataset, license, model, provider, date, commit, and the command that
+  produces it; `docs/BENCHMARKS.md` says what running that command needs.
 - Keep changes focused and update ExDoc or Livebooks with public API changes.
 
 ## Pull Requests

README.md

@@ -178,13 +178,21 @@ protocol endpoint. See [protocol integration and migration](docs/PRODUCTION_OPER
 - [Production Operations](docs/PRODUCTION_OPERATIONS.md) — credentials,
   telemetry, concurrency, persistence, failure handling, and the protocol
   adapters.
+- [Ticket Routing Tutorial](docs/TUTORIAL_TICKET_ROUTING.md) — score a router
+  on held-out data, improve it with an optimizer, and prove the improvement on
+  tickets it has never seen. About a cent to run yourself.
 - [Runnable Livebooks](livebooks/01_real_lm_front_door.livemd) — inspect the
   same progression in IEx-ready notebooks.
+- [Benchmarks](https://github.com/deepfates/imp/blob/main/docs/BENCHMARKS.md) — every number this repository publishes,
+  the exact command that produces it, what that command costs you, and what
+  cannot be re-measured at all. The numbers themselves are one row each in
+  [benchmarks/RESULTS.md](https://github.com/deepfates/imp/blob/main/benchmarks/RESULTS.md).
+- [Case study: GEPA and MIPROv2 on TREC](https://github.com/deepfates/imp/blob/main/docs/CASE_STUDY_TREC.md) — a matched
+  optimizer comparison against pinned DSPy, recomputable from committed rows
+  but not reproducible, and labeled that way.
 - [Evidence](https://github.com/deepfates/imp/blob/main/docs/EVIDENCE.md) —
-  the research side: how far the DSPy parity aim has actually been
-  established, benchmark reproductions, execution traces and portable
-  trajectories, and what remains a target. It lives in the source repository,
-  not in the installed package.
+  what kind of evidence stands behind which kind of claim, and the record of
+  the runs that did not work.
 
 Run `mix docs` for the exhaustive module and function reference.