deepfates · deepfates · Sep 17, 2026 · Sep 17, 2026 · Sep 17, 2026
@@ -202,7 +202,7 @@ jobs:
       # Its self-test imports gepa_artifact from the checkout root; the
       # maintainer environment had that on the ambient path, runners don't.
       - run: PYTHONPATH="$PWD/tmp/gepa-artifact" scripts/setup_corrected_gepa_comparator.sh tmp/gepa-artifact
-      # Source-exact IFBench scorer venv (docs/internal/BENCHMARK_TRUTH.md).
+      # Source-exact IFBench scorer venv (docs/BENCHMARKS.md).
       - run: |
           python3 -m venv tmp/ifbench-parity-venv
           tmp/ifbench-parity-venv/bin/python -m pip install -q -r benchmarks/requirements-ifbench-parity.txt
@@ -213,10 +213,11 @@ jobs:
       # spend their test timeout compiling.
       # Children run in both dev (banking77) and test (hotpotqa) envs.
       - run: cd examples/deployment && IMP_PATH=../.. mix deps.get && IMP_PATH=../.. mix deps.compile && IMP_PATH=../.. mix compile && IMP_PATH=../.. MIX_ENV=test mix deps.compile && IMP_PATH=../.. MIX_ENV=test mix compile
-      # Known-red residue at introduction (tracked on imp-sqkr): the sealed
-      # v1 contract detects mix.lock drift after the bandit CVE bump, and the
-      # MuSiQue receipts read a machine-local /tmp data dir. This job makes
-      # that visible on every run rather than hiding it in the fast lane.
+      # Re-measured 2026-09-17 on a clean tracked tree: neither residue this
+      # comment used to describe is still red. The mix.lock drift is gone and
+      # the MuSiQue receipts are excluded by their own tag. The only remaining
+      # failures came from Imp.UpstreamFidelityTest, which is removed with the
+      # rest of the conformance-ledger machinery.
       - run: mix differential.check
 
   docs:

@@ -62,6 +62,9 @@ ticket file in this repository.
 - Preserve upstream algorithmic semantics when using an upstream name.
 - Give deliberate Elixir-native alternatives a distinct contract and rationale.
 - Never turn fixtures, smoke tests, or symbol presence into broad parity claims.
+- Every published number needs a row in `benchmarks/RESULTS.md` carrying its
+  dataset, license, model, provider, date, commit, and the command that
+  produces it; `docs/BENCHMARKS.md` says what running that command needs.
 - Keep changes focused and update ExDoc or Livebooks with public API changes.
 
 ## Pull Requests

@@ -189,13 +189,21 @@ protocol endpoint. See [protocol integration and migration](docs/PRODUCTION_OPER
 - [Production Operations](docs/PRODUCTION_OPERATIONS.md) — credentials,
   telemetry, concurrency, persistence, failure handling, and the protocol
   adapters.
+- [Ticket Routing Tutorial](docs/TUTORIAL_TICKET_ROUTING.md) — score a router
+  on held-out data, improve it with an optimizer, and prove the improvement on
+  tickets it has never seen. About a cent to run yourself.
 - [Runnable Livebooks](livebooks/01_real_lm_front_door.livemd) — inspect the
   same progression in IEx-ready notebooks.
+- [Benchmarks](https://github.com/deepfates/imp/blob/main/docs/BENCHMARKS.md) — every number this repository publishes,
+  the exact command that produces it, what that command costs you, and what
+  cannot be re-measured at all. The numbers themselves are one row each in
+  [benchmarks/RESULTS.md](https://github.com/deepfates/imp/blob/main/benchmarks/RESULTS.md).
+- [Case study: GEPA and MIPROv2 on TREC](https://github.com/deepfates/imp/blob/main/docs/CASE_STUDY_TREC.md) — a matched
+  optimizer comparison against pinned DSPy, recomputable from committed rows
+  but not reproducible, and labeled that way.
 - [Evidence](https://github.com/deepfates/imp/blob/main/docs/EVIDENCE.md) —
-  the research side: how far the DSPy parity aim has actually been
-  established, benchmark reproductions, execution traces and portable
-  trajectories, and what remains a target. It lives in the source repository,
-  not in the installed package.
+  what kind of evidence stands behind which kind of claim, and the record of
+  the runs that did not work.
 
 Run `mix docs` for the exhaustive module and function reference.
 

@@ -0,0 +1,86 @@
+# Results
+
+One row per published number. Every row carries the dataset it was measured on,
+the dataset's license, the model and provider that produced it, the date, the
+commit, and the command. Prose elsewhere in this repository cites a row here
+rather than restating a number.
+
+A row is **re-measurable** if a stranger with an API key can run its command and
+get a comparable number, and **recomputable** if the command only recomputes
+statistics from committed rows. The distinction is not cosmetic: a recomputable
+row cannot tell you whether the original measurement was made correctly, only
+that the arithmetic on the retained rows is what the doc says it is.
+
+See [docs/BENCHMARKS.md](../docs/BENCHMARKS.md) for what each command needs and
+for the claims that cannot be re-measured at all.
+
+## Re-measurable
+
+| # | Number | Dataset (license) | Model | Provider | Date | Commit | Command |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| R1 | Zero-shot held-out accuracy `0.30`–`0.50` over 3 repeats (20 held-out of 60 tickets) | `priv/tutorial/support_tickets.json`, 60 rows, sha256 `7ea5ae7a…` (written for this repository; see [SUPPORT_TICKETS_LICENSE.md](../priv/tutorial/SUPPORT_TICKETS_LICENSE.md)) | `gpt-5.4-mini` | OpenRouter route `openai/gpt-5.4-mini` | 2026-08-22 | `88d61a9c` | `OPENAI_API_KEY=… mix run scripts/tutorial_ticket_routing_experiment.exs` |
+| R2 | `LabeledFewShot(k: 8)` held-out accuracy `0.95`–`1.00` over the same 3 repeats; per-repeat lift `+0.45`, `+0.65`, `+0.65` | same as R1 | `gpt-5.4-mini` | OpenRouter route `openai/gpt-5.4-mini` | 2026-08-22 | `88d61a9c` | same as R1 |
+
+R1 and R2 come from one execution of one command; they are two numbers from the
+same three repeats, not independent measurements. That run used 120 requests,
+44,365 tokens and `$0.038819` in provider-priced usage for all three repeats —
+about `$0.013` and 8–9 seconds per repeat. No row errored and the in-BEAM cache
+was cleared before each repeat, so all 120 calls were live.
+
+Three repeats of a twenty-row evaluation is a coarse instrument. The gap between
+the two rows (45–65 points) is far larger than the instrument's resolution
+(one row is 5 points), which is why the direction is trustworthy while the exact
+endpoints are not.
+
+## Recomputable only
+
+| # | Number | Dataset (license) | Model | Provider | Date | Commit | Command |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| R3 | Imp GEPA minus its own baseline, held-out accuracy `+0.4000`, 95% CI `[0.2958, 0.5042]`, Holm-adjusted `p = 0.00020` | TREC fine-grained, 20 train / 40 selection / 80 held-out drawn from `benchmarks/data/confidence-calibration-trec-fine.jsonl` (see [TREC_ATTRIBUTION.md](data/TREC_ATTRIBUTION.md)) | task `gpt-5.4-mini`, optimizer `claude-sonnet-4.6` | OpenRouter | 2026-07-26 | sealed in `examples/matched_instruction_optimizers_trec/contract.json` | `mix run --no-start examples/matched_instruction_optimizers_trec/recompute_compact.exs -- examples/matched_instruction_optimizers_trec/contract.json examples/matched_instruction_optimizers_trec/data/imp-scored-rows.json examples/matched_instruction_optimizers_trec/data/upstream-scored-rows.json examples/matched_instruction_optimizers_trec/data/aggregate-recomputed.json` |
+| R4 | Imp MIPROv2 minus its own baseline, held-out accuracy `+0.1458`, 95% CI `[0.0458, 0.2458]`, Holm-adjusted `p = 0.00270` | same as R3 | same as R3 | OpenRouter | 2026-07-26 | same as R3 | same as R3 |
+| R5 | Imp GEPA minus pinned DSPy 3.2.1 GEPA, held-out accuracy `-0.0083`, 95% CI `[-0.0458, 0.0292]`, above the preregistered `-0.05` noninferiority margin | same as R3 | same as R3 | OpenRouter | 2026-07-26 | same as R3 | same as R3 |
+
+R3, R4 and R5 are three seeds (`2026072602`, `2026072603`, `2026072604`) of one
+sealed experiment that used 6,491 model calls and `$3.13862325`. The command
+replays gold-label checks, row scoring, source-clustered bootstrapping, Holm
+correction and the noninferiority decision from the committed scored rows. It
+does not re-contact a provider. The raw request-level traces (181 MB) were not
+published, so nobody outside this repository can check that the committed rows
+are what the providers actually returned. See
+[docs/CASE_STUDY_TREC.md](../docs/CASE_STUDY_TREC.md).
+
+| # | Number | Dataset (license) | Model | Provider | Date | Commit | Command |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| R6 | Optimize Anything on three ReActV2 tool descriptions: held-out mean over 4 untouched requests `0.95` → `1.0` | 4 held-out agent requests written for `examples/deployment` (see [examples/deployment/data/README.md](../examples/deployment/data/README.md)) | task `gpt-5.4-mini`, reflection `claude-sonnet-4.6` | OpenRouter | 2026-08-23 | `8a6ce8fd` | `mix test test/deployment_agent_optimization_example_test.exs` verifies the retained result; reproducing it needs `OPENROUTER_API_KEY` and `examples/deployment/agent_optimization.exs` |
+
+R6 used 72 task requests for `$0.054251` and 3 reflection requests for
+`$0.010494`, each under a separate one-dollar hard cap. It is one stochastic
+treatment over four held-out requests: a move from 19/20 to 20/20 scoring
+points. It establishes that the component-optimization, action-observation,
+Artifact and restart path runs end to end. It does not establish agent
+effectiveness. The retained result and Artifact are under
+`examples/deployment/evidence/`.
+
+## Findings that are not results
+
+These are recorded because deleting them would misrepresent the record. None of
+them can be re-measured from this repository: the raw artifacts that diagnosed
+them are not all published, and in two cases the machinery that produced them
+has since changed. They are dated observations, not standing claims.
+
+The first three rows below have no recoverable run date. Their artifacts lived
+under `benchmarks/results/`, which is not tracked, and no commit in this
+repository's history ever contained them — so the only date that can be stated
+honestly is the date the finding entered the record, `2026-08-09` in commit
+`bbb2a983`, when the verdicts were written down from artifacts that were then
+still on a maintainer's disk.
+
+| Finding | Date | What was observed |
+| --- | --- | --- |
+| HotPotQA JSON-GEPA, mean lift `-0.015` over 3 seeds (task `gpt-5.4-mini`, reflection `claude-sonnet-4.6`) | run date not recorded; recorded `2026-08-09` in `bbb2a983` | A completed treatment that did not improve held-out performance. Its 32 semantic metric calls were tiny beside the GEPA artifact's 6,871 for HotpotQA; the 24-row test moves in 0.042 steps; six strict-adapter parse failures scored zero on one seed. |
+| Banking77 modeled-MIPRO, two conditions at `+0.0417` and `+0.0208` (2 of 3 improving seeds each) | run date not recorded; recorded `2026-08-09` in `bbb2a983` | Both missed the preregistered `≥0.05` bar. Proposals, attached demos and acquisitions were real. The 48-row test and a high baseline explain the resolution limit. |
+| Grue stateful-agent GEPA, 0 of 3 seeds improved (local `llama3.2:3b`) | run date not recorded; recorded `2026-08-09` in `bbb2a983` | Every candidate scored 0.0 on every selection row, so the optimizer had no ranking signal and retained the baseline. The treatment could not answer the question it was posed. |
+| IFBench scorer defect | fixed `2026-07-31` in `8c798d2e`; recorded `2026-08-09` in `bbb2a983` | The scorer represented nested rule arguments incorrectly and used a non-pinned language fallback. Optimizer results produced with the faulty scorer are invalid and were withdrawn, not rescored. |
+| Matched IFBench 16k rehearsal | recorded `2026-08-20` in `6ff77214` | Both runtimes sealed six optimization-and-selection cells at 16384-token settings for `$16.42`, then the campaign stopped in the held-out phase on a 4096-token input bound of our own. It produced no held-out verdict. An earlier draft compared imp `0.8542` against upstream `0.8698`; those are different quantities (an internal champion score versus an independent re-scoring) and the comparison is withdrawn in full. |
+| ChatAdapter parse failures dominate per-cell variation | recorded `2026-08-20` in `6ff77214` | Across takes of the above, upstream logged 0–5 parse failures per 32-row evaluation (mean 8.3%), correlating with the take's mean at `r = -0.84`. Any future outcome on this design must be reported as two numbers, parse rate and score-given-parse. |
+| GEPA Pareto pruning divergence | recorded `2026-08-10` in `ba3311df` | Found by the recorded-tape GEPA differential: ties were broken by an Elixir term-printing artifact rather than upstream's stable discovery order. A score comparison at the power above could not have detected it. This one is still caught on every run by `mix differential.check`, which is free and has no sampling noise. |
@@ -0,0 +1,25 @@
+# GEPA six-family splits attribution
+
+`gepa-campaign-full/families.json` pins the six families of the GEPA paper's
+benchmark — AIMEBench, HotpotQABench, hoverBench, IFBench, LiveBenchMathBench
+and Papillon — by family name, program shape, signature, upstream metric, split
+row counts and a SHA-256 for each of the train, dev and test splits.
+
+**The split files themselves are not in this repository.** `families.json` is a
+manifest of data that is absent: nothing under `benchmarks/data/` matches those
+checksums, for any of the six families. Any claim that rests on running these
+splits therefore cannot be re-measured here. See the "Cannot be re-measured"
+section of [docs/BENCHMARKS.md](../../docs/BENCHMARKS.md).
+
+Upstream source, as recorded in the manifest:
+
+- GEPA artifact repository, commit `cbefbc1aa0f43dd39874ec4bf42211365dbda42e`:
+  https://github.com/gepa-ai/gepa-artifact
+- GEPA paper: https://arxiv.org/abs/2507.19457
+
+The underlying corpora have their own licenses, which differ per family:
+HotpotQA is CC BY-SA 4.0, HoVer derives from HotpotQA and Wikipedia, and
+Papillon uses the PUPA corpus. AIMEBench, IFBench and LiveBenchMathBench are
+**unknown**: the artifact repository does not declare a license for them and we
+have not resolved one. The manifest's `dataset_aliases` name the Hugging Face
+repositories where two of the six can be obtained.
@@ -0,0 +1,37 @@
+# IFBench rows: what is and is not IFBench here
+
+Two different things in this repository carry the name IFBench. They have
+different provenance and only one of them is the benchmark.
+
+## `ifbench_instruction_following` — not IFBench data
+
+`mix imp.benchmark.fetch --tasks ifbench_instruction_following` writes
+`ifbench_instruction_following-test-0-N.jsonl` into the output directory. Those
+rows are **written in this repository**, in
+`bench/imp/benchmark_truth/fetcher.ex`, under the internal dataset name
+`imp/local-ifbench`, config `verifier-smoke`. They are a handful of
+hand-authored instruction/constraint pairs that exercise the constraint
+verifier. They are MIT, like the rest of this repository.
+
+They are not drawn from IFBench, they are not a sample of it, and a score on
+them is not an IFBench score. Naming the task after the benchmark it was
+modeled on was a mistake that this note exists to correct until the name
+changes.
+
+## The matched IFBench experiments — real IFBench, license unknown
+
+`examples/matched_gepa_mipro_ifbench*/` and
+`examples/matched_ifbench_rehearsal16k/` use the real IFBench task graph,
+metric and splits (`IFBench_train[0:300]`, `IFBench_test`), taken from the GEPA
+artifact repository:
+
+- https://github.com/gepa-ai/gepa-artifact
+- GEPA paper: https://arxiv.org/abs/2507.19457
+- IFBench scorer dependencies are pinned in
+  `benchmarks/requirements-ifbench-parity.txt`
+
+**License: unknown.** The artifact repository declares no license for the
+IFBench data, and we have not resolved one with its authors. The corpus files
+themselves are not committed here; the experiment contracts pin them by digest
+and the harness materializes them. The repository's MIT license makes no
+statement about that corpus.
@@ -0,0 +1,27 @@
+# TREC question-classification subset attribution
+
+`confidence-calibration-trec-fine.jsonl` is a 400-row materialization (200
+calibration, 200 held-out) of the TREC question-classification corpus with
+fine-grained labels. `benchmarks/data/confidence-calibration-trec-fine.provenance.json`
+records the two source files, their SHA-256 digests, the deduplication and
+overlap rules, and the selection seed;
+`benchmarks/data/build_confidence_calibration_trec.py` rebuilds it.
+
+The matched GEPA/MIPROv2 experiment in
+`examples/matched_instruction_optimizers_trec/` draws its 20 train, 40
+selection and 80 held-out rows from this file. Those splits are listed by
+source id in that example's `contract.json`.
+
+Source:
+
+- Project: https://cogcomp.seas.upenn.edu/Data/QA/QC/
+- Files: `train_5500.label`, `TREC_10.label`
+- Dataset card: https://huggingface.co/datasets/CogComp/trec
+- Papers: Li and Roth, COLING 2002; Hovy et al., HLT 2001
+
+**License: unknown.** The pinned dataset card states no license, and the
+project page distributes the label files without one. The provenance file
+records this as `"unknown (as reported by the pinned dataset card)"` rather
+than guessing. The repository's MIT license covers our code and our derived
+split files; it makes no statement about the underlying corpus. If you intend
+to redistribute these rows, resolve the license with the corpus authors first.