diff --git a/CHANGELOG.md b/CHANGELOG.md index 965a8fac..e5544bea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,17 @@ User-visible changes to Imp are recorded here. ## Unreleased +## 0.4.0 — 2026-09-17 + +- `Imp.ACP` and `Imp.MCP.connect/2` are part of Imp. The separate `imp_acp` + package is retired: `Imp.ACP.start_link/1` and `Imp.ACP.run/1` expose an + ordinary Imp program to an ACP host, and `Imp.MCP.connect/2` imports + authorized MCP servers through ExMCP as ordinary tools with explicit + connection cleanup. There is no compatibility shim; a consumer that depended + on `imp_acp` depends on `imp` alone and changes the module prefix. ExMCP is + declared `runtime: false`, so an OTP release that uses either must list + `applications: [ex_mcp: :load]` in its release definition. Ordinary Imp + startup still starts no protocol endpoint. - A map or list value in a prompt, including a structured tool result, now renders the way DSPy renders a dict: `json.dumps(..., ensure_ascii=False)` with Python's default separators, complete. It was `inspect/1` at its @@ -88,6 +99,24 @@ User-visible changes to Imp are recorded here. `{:mcp_tools_list_failed, server, {:invalid_mcp_tools_response, shape}}` rather than as a bare `{:invalid_mcp_tools_response, shape}` that named no server. +- The repository is public. Installing at a tag needs no credentials, and the + README, docs, and livebooks no longer describe a private source release. +- Removed the evidence-certification bookkeeping from the source checkout. It + never shipped in the package, so a consumer sees no change; the benchmark + harness it wrapped is unchanged. +- Added [Benchmarks](https://github.com/deepfates/imp/blob/main/docs/BENCHMARKS.md) + and its [results table](https://github.com/deepfates/imp/blob/main/benchmarks/RESULTS.md). + Every number this repository publishes is one row in that table, carrying + the dataset and its license, the model, the provider, the date, the commit, + and the command that produced it; prose elsewhere cites a row rather than + restating a number. The benchmarks page says what each command needs from + you — key, Python environment, time, rough cost — and separates a row a + stranger can re-measure with an API key from one that only recomputes + statistics from committed rows, and from the claims that cannot be + re-measured at all. Neither page publishes an aggregate or a release score. + The tutorial's rows were re-measured live for this release, a month after the + first run, and both runs are recorded. They live in the repository, not in + the installed package. ## 0.3.2 — 2026-09-01 diff --git a/README.md b/README.md index 56054a58..b725dbc0 100644 --- a/README.md +++ b/README.md @@ -6,12 +6,6 @@ ordinary OTP application. It brings the central idea of [DSPy](https://dspy.ai)—programming behavior and improving it from examples rather than hand-editing prompts—to the BEAM. -**Release state.** The released tag `v0.3.2` does not contain `Imp.ACP`. The -integrated protocol adapters — `Imp.ACP` and `Imp.MCP.connect/2`, absorbed from -the retired `imp_acp` package — exist only on `main` until the next tag. -Installing `v0.3.2` gives you the library described below without them; using -the adapters today means depending on `main` (see [Install](#install)). - Here, “typed” means required inputs are checked and model outputs are parsed and validated against the signature before application code receives them. For DSPy compatibility, a supplied input whose value disagrees with its @@ -135,13 +129,9 @@ Imp is not published to Hex. It is installed from this public repository, at an immutable tag, with no credentials: ```elixir -{:imp, github: "deepfates/imp", tag: "v0.3.2"} +{:imp, github: "deepfates/imp", tag: "v0.4.0"} ``` -That tag does not contain `Imp.ACP` or `Imp.MCP.connect/2`; a tag that does -will be cut. Until then, depend on `{:imp, github: "deepfates/imp", branch: -"main"}` if you need the adapters. - ExMCP is declared `runtime: false`, so an OTP release that uses `Imp.ACP` or `Imp.MCP` must list `applications: [ex_mcp: :load]` in its release definition; see [protocol runtime in releases](docs/PRODUCTION_OPERATIONS.md#protocol-runtime-in-releases). @@ -152,9 +142,9 @@ modules are ported from. Use `{:imp, path: "path/to/imp"}` only while developing against a local checkout. Imp requires Elixir `~> 1.19`. Commit your application's `mix.lock`; the Git tag fixes Imp's source, while normal Mix constraints may otherwise -resolve newer compatible transitive versions. Version `0.3.2` contains the -breaking `0.3` changes from `0.2.1`; see the [release notes](RELEASE_NOTES.md) -when upgrading. +resolve newer compatible transitive versions. Version `0.4.0` contains +breaking changes from `0.3.2`; see the [release notes](RELEASE_NOTES.md) when +upgrading. Imp uses [ReqLLM](https://hex.pm/packages/req_llm) for model providers. The examples use OpenAI, but programs are not tied to that provider. The @@ -164,9 +154,6 @@ provider-free parts of the learning path and deployment example do too. ## Connect tools or expose a program -Everything in this section is on `main` only, not in `v0.3.2` (see the release -state at the top). - `Imp.MCP.connect/2` imports authorized MCP servers through ExMCP, returning ordinary tools plus explicit connection cleanup. Source server/tool identities, schemas, and annotations remain in each tool's `metadata.mcp` even when names @@ -209,8 +196,8 @@ Run `mix docs` for the exhaustive module and function reference. For an ordinary ACP workspace agent with bounded tools, see [examples/workspace_agent](examples/workspace_agent/README.md). It depends -directly on this Imp checkout (`main` only, like the adapters it uses) and -includes a provider-free mode for checking its launcher and workspace boundary. +directly on this Imp checkout by path and includes a provider-free mode for +checking its launcher and workspace boundary. ## Where this fits diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index fc3be01f..02caffdc 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,101 +1,78 @@ -# Imp v0.3.2 +# Imp v0.4.0 Imp is a framework for typed, optimizable language-model programs on the BEAM. Declare a task as named inputs and outputs, call it like any other Elixir program, measure it on examples, compile it with an optimizer, and run the selected program under OTP. +This release absorbs the protocol adapters that previously lived on `main` +only, and changes three published shapes. It is `0.4.0` rather than a patch +because a program written against `v0.3.2` can need edits. + ## Install -`v0.3.2` is a Git source release from a public repository; no credentials are +`v0.4.0` is a Git source release from a public repository; no credentials are required. ```elixir -{:imp, github: "deepfates/imp", tag: "v0.3.2"} +{:imp, github: "deepfates/imp", tag: "v0.4.0"} ``` Imp is not published to Hex. Use a path dependency only while developing against a local checkout. -This patch release makes the RLM controller's language guide match its -restricted interpreter and preserves explicit zero-retry provider policy for -all Req adapter forms, including MFA adapters with arguments. - -## What is included - -- Typed signatures with scalar, collection, enum, union, optional, default, - code, and constrained fields. -- `Predict`, `ChainOfThought`, composed `Imp.Module` programs, retrieval, - ReActV2, CodeAct, RLM, tools, MCP, and provider streaming. -- Examples, metrics, concurrent evaluation, disjoint train/selection/test - experiments, and optimizer reports. -- Demonstration, instruction, prompt, ensemble, rule, playbook, and - weight-training optimizer families, including GEPA, MIPROv2, SIMBA, COPRO, - BootstrapFewShot, RandomSearch, KNNFewShot, BootstrapFinetune, BetterTogether, - Avatar, and Optimize Anything. -- Checksummed whole-program and parameter artifacts that exclude credentials - and apply selected state to freshly constructed trusted code. -- OTP-native operation with bounded tasks, cancellation, per-effect - authorization, redacted telemetry, caching, usage accounting, hot reload, - and failure propagation. -- ReqLLM provider clients, explicit local/static test models, retriever and - trainer extension points, and local MLX/TRL integration boundaries. - -The [Learning Path](docs/LEARNING_PATH.md) builds one program from its first -provider call through evaluation, optimization, tools, persistence, and -deployment. The [deployment example](examples/deployment/README.md) shows a -supervised two-stage program with parameter reload, concurrent calls, restart, -timeouts, and crash containment. Five Livebooks cover the same system -interactively. - -## What the BEAM changes - -Imp preserves DSPy's program/evaluate/optimize workflow without copying -Python's object model. Programs are immutable values. Configuration can be -explicit or process-scoped. Evaluation and tool work run in supervised tasks. -Telemetry uses standard `:telemetry` events. Saved state is rebound to live -providers and callbacks at application startup instead of serializing runtime -authority. - -Provider output and optimizer search are stochastic. A compiled program is a -candidate until it improves the metric that matters on data excluded from -training and selection. Imp supplies that lifecycle; applications still own -their data, metric, budget, promotion rule, and operational policy. - -DSPy's Python integration ecosystem is larger. Imp exposes extension points -for providers, retrievers, adapters, tools, and trainers, but Python-only -integrations do not automatically work on the BEAM. DSPy's Flex code optimizer -is not included in this release. - -The supported center is the `Imp` facade, signatures, adapters, evaluation, -static and ReqLLM execution, tools, telemetry, saving, and the deployment -pattern. Generated docs place optimizer implementations, parameter artifacts, -agent loops, training integrations, and `Imp.Run` in **Experimental optimizers -and advanced workflows**. These are implemented and tested APIs, not release -promises of effectiveness or pre-1.0 shape stability. In particular, GRPO is -an external-training boundary rather than an in-process gradient engine. - -## Breaking changes from v0.2.1 - -- `Imp.optimize/3`, `/4`, and `/5` return `{:ok, program}` or - `{:error, reason}`. Use the corresponding `Imp.optimize!` function when a - failure should raise. -- `Imp.Adapters.Types` and its nested structs moved to `Imp.Adapter.Types`. -- `Imp.Agent` and `Imp.Agent.Runtime` were removed. Use ReActV2 or RLM as the - program and ordinary Elixir supervision as the runtime. Use - `Imp.start_run/3` only when a host needs ordered events, addressable - cancellation, or explicit effect authorization. +ExMCP is declared `runtime: false`, so an OTP release that uses `Imp.ACP` or +`Imp.MCP` must list `applications: [ex_mcp: :load]` in its release +definition; see [protocol runtime in +releases](docs/PRODUCTION_OPERATIONS.md#protocol-runtime-in-releases). +Ordinary Imp startup starts no protocol endpoint. + +## Headline changes + +- `Imp.ACP` and `Imp.MCP.connect/2` are in the tag. The separate `imp_acp` + package is retired with no compatibility shim: a consumer that depended on + it now depends on `imp` alone. `Imp.MCP.connect/2` also gains OAuth + credentials for remote HTTP servers (`Imp.MCP.OAuth`), `bearer_env` + descriptor auth, `on_failure: :drop` with an `unavailable` list, and a + per-dial timeout. +- `:reasoning_effort` is the one reasoning option on `Imp.Clients.ReqLLM`. + `:openrouter_reasoning` is gone; the wire encoding is the separate + `:openrouter_reasoning_wire`. +- ReActV2 sends the tool roster natively and no longer declares a `tools` + input field or writes its instructions into `signature.instructions`. Loop + guidance travels to the adapter through `:adapter_opts`. +- The `:model_response` event's `metadata.cost` is a plain USD float or `nil`, + with any provider breakdown under `metadata.billing`. +- Structured values in a prompt render complete, the way DSPy renders a dict, + instead of a truncated `inspect/1`. + +## Breaking changes from v0.3.2 + +- Replace `openrouter_reasoning: ...` with `reasoning_effort: ...`. Saved + programs allowlist `:reasoning_effort` and `:openrouter_reasoning_wire` in + its place, so rebuild artifacts that carried the old key. +- A caller that passed or read ReActV2's `tools` input field no longer has + one; the roster is sent natively. +- A host that read `metadata.cost` as a provider billing map reads a number + now, and finds the map under `metadata.billing` when the provider sent one. ## Upgrade path -1. Replace `Imp.Adapters.Types` references with `Imp.Adapter.Types`. -2. Choose the returning or raising optimizer API explicitly. -3. Replace `Imp.Agent` usage with a ReActV2/RLM program owned by your - supervision tree. -4. Rebuild saved artifacts with `0.3.2` before promotion. +1. Rename the reasoning option and rebuild saved artifacts with `0.4.0`. +2. Drop any `tools` handling around ReActV2. +3. Sum spend from `metadata.cost` as a number. +4. If you depended on `{:imp, github: "deepfates/imp", branch: "main"}` for the + adapters, move to the tag. 5. Run your held-out evaluation and application smoke test against the tagged dependency. -Generated module documentation is the complete API reference. Start with -`Imp`, `Imp.Signature`, `Imp.Module`, `Imp.Evaluate`, `Imp.Optimizer`, -`Imp.Optimizer.Artifact`, `Imp.Run`, and `Imp.Telemetry`. +New in this release: [Benchmarks](https://github.com/deepfates/imp/blob/main/docs/BENCHMARKS.md) +and its [results table](https://github.com/deepfates/imp/blob/main/benchmarks/RESULTS.md) +carry every number this repository publishes with the command that produces it, +and the ticket-routing rows were re-measured live for this release, a month +after the first run, with both runs recorded. + +The [CHANGELOG](CHANGELOG.md) records every user-visible change in this +release. Generated module documentation is the complete API reference. Start +with `Imp`, `Imp.Signature`, `Imp.Module`, `Imp.Evaluate`, `Imp.Optimizer`, +`Imp.ACP`, `Imp.MCP`, and `Imp.Telemetry`. diff --git a/benchmarks/RESULTS.md b/benchmarks/RESULTS.md index 02fea676..8d0b130b 100644 --- a/benchmarks/RESULTS.md +++ b/benchmarks/RESULTS.md @@ -18,19 +18,36 @@ for the claims that cannot be re-measured at all. | # | Number | Dataset (license) | Model | Provider | Date | Commit | Command | | --- | --- | --- | --- | --- | --- | --- | --- | -| R1 | Zero-shot held-out accuracy `0.30`–`0.50` over 3 repeats (20 held-out of 60 tickets) | `priv/tutorial/support_tickets.json`, 60 rows, sha256 `7ea5ae7a…` (written for this repository; see [SUPPORT_TICKETS_LICENSE.md](../priv/tutorial/SUPPORT_TICKETS_LICENSE.md)) | `gpt-5.4-mini` | OpenRouter route `openai/gpt-5.4-mini` | 2026-08-22 | `88d61a9c` | `OPENAI_API_KEY=… mix run scripts/tutorial_ticket_routing_experiment.exs` | -| R2 | `LabeledFewShot(k: 8)` held-out accuracy `0.95`–`1.00` over the same 3 repeats; per-repeat lift `+0.45`, `+0.65`, `+0.65` | same as R1 | `gpt-5.4-mini` | OpenRouter route `openai/gpt-5.4-mini` | 2026-08-22 | `88d61a9c` | same as R1 | +| R1 | Zero-shot held-out accuracy `0.30`–`0.40` over 3 repeats (20 held-out of 60 tickets) | `priv/tutorial/support_tickets.json`, 60 rows, sha256 `7ea5ae7a…` (written for this repository; see [SUPPORT_TICKETS_LICENSE.md](../priv/tutorial/SUPPORT_TICKETS_LICENSE.md)) | `gpt-5.4-mini` | OpenRouter route `openai/gpt-5.4-mini` | 2026-09-17 | `7985ed2f` | `OPENAI_API_KEY=… mix run scripts/tutorial_ticket_routing_experiment.exs` | +| R2 | `LabeledFewShot(k: 8)` held-out accuracy `0.90`–`0.95` over the same 3 repeats; per-repeat lift `+0.55`, `+0.55`, `+0.65` | same as R1 | `gpt-5.4-mini` | OpenRouter route `openai/gpt-5.4-mini` | 2026-09-17 | `7985ed2f` | same as R1 | R1 and R2 come from one execution of one command; they are two numbers from the same three repeats, not independent measurements. That run used 120 requests, -44,365 tokens and `$0.038819` in provider-priced usage for all three repeats — -about `$0.013` and 8–9 seconds per repeat. No row errored and the in-BEAM cache -was cleared before each repeat, so all 120 calls were live. +44,293 tokens and `$0.038488` in provider-priced usage for all three repeats — +about `$0.013` and 8–13 seconds per repeat. No row errored and the in-BEAM cache +was cleared before each repeat, so all 120 calls were live. The run's artifact +is committed at +[`benchmarks/data/tutorial-ticket-routing-2026-09-17.receipt.json`](data/tutorial-ticket-routing-2026-09-17.receipt.json), +sha256 `2dcc1228…`; the dollar figure in it is the script's own pricing table +(`$0.75`/`$4.50` per million tokens) applied to reported token counts, not a +provider-billed amount, so treat it as an estimate and the token counts as the +measurement. + +The same command was run a month earlier, on 2026-08-22 at commit `88d61a9c`, +against the same dataset, model and route: zero-shot `0.30`–`0.50`, optimized +`0.95`–`1.00`, per-repeat lift `+0.45`, `+0.65`, `+0.65`, 120 requests and +`$0.038819` in provider-priced usage. Two independent runs a month apart agree +on the thing worth claiming — every repeat improved, by 45 to 65 points — and +disagree on the endpoints, which is what a twenty-row evaluation should do. Three repeats of a twenty-row evaluation is a coarse instrument. The gap between -the two rows (45–65 points) is far larger than the instrument's resolution -(one row is 5 points), which is why the direction is trustworthy while the exact -endpoints are not. +the two rows (55–65 points on 2026-09-17, 45–65 points on 2026-08-22) is far +larger than the instrument's resolution (one row is 5 points), which is why the +direction is trustworthy while the exact endpoints are not. The 2026-09-17 run +put one optimized repeat at `0.90`, below the `0.95`–`1.00` the tutorial claimed +from the first run alone, and one repeat at 13.1 seconds, above the 8–9 seconds +it claimed; both claims were widened to the measured union rather than restated +from the luckier run. ## Recomputable only diff --git a/benchmarks/data/tutorial-ticket-routing-2026-09-17.receipt.json b/benchmarks/data/tutorial-ticket-routing-2026-09-17.receipt.json new file mode 100644 index 00000000..070aaab4 --- /dev/null +++ b/benchmarks/data/tutorial-ticket-routing-2026-09-17.receipt.json @@ -0,0 +1,209 @@ +{ + "configuration": { + "adapter": "Imp.Adapter.JSON", + "cache_cleared_per_repeat": true, + "json_retries": 1, + "k": 8, + "max_concurrency": 8, + "metric": "Imp.exact_match(:team)", + "optimizer": "Imp.Optimizer.LabeledFewShot", + "repeats": 3, + "signature": "ticket -> team: enum[atlas,harbor,beacon,quill]", + "timeout_ms": 60000, + "trainset": "train split only; dev split unused; test split held out" + }, + "dataset": { + "dev": 20, + "path": "priv/tutorial/support_tickets.json", + "sha256": "7ea5ae7ad724dc4311cf954bafeb48a4b7b284b619c33bc00b30d37ad6df6f87", + "test": 20, + "train": 20 + }, + "doc_claims_under_test": { + "baseline_repeat_range": [ + 0.3, + 0.5 + ], + "cost_usd_per_repeat_approx": 0.013, + "duration_seconds_range": [ + 8, + 9 + ], + "optimized_repeat_range": [ + 0.95, + 1.0 + ], + "source": "docs/TUTORIAL_TICKET_ROUTING.md" + }, + "fresh_service": { + "artifact_sha256": "cacd6fb5f6538da5fb7a3e7569e052e4735cafb8b404ab4e6aea3dda92537434", + "budget": { + "active_reservations": 0, + "exhausted": null, + "limits": { + "input_tokens": 50000, + "output_tokens": 1024, + "requests": 4, + "usd": 0.1 + }, + "pricing": { + "input_per_million": 0.75, + "output_per_million": 4.5, + "source_url": "https://openai.com/api/pricing/" + }, + "requests": 4, + "reservation_ledger_version": 2, + "reservations": [], + "reserved": { + "input_tokens": 0, + "output_tokens": 0, + "usd": 0.0 + }, + "single_attempt_transport_enforced": true, + "transport_attempts": 4, + "usage": { + "input_tokens": 1987, + "output_tokens": 55, + "usd": 0.0017360000000000001 + } + }, + "concurrency": 4, + "correct": 4, + "fresh_os_process": true, + "total": 4 + }, + "generated_at": "2026-09-17T20:14:18Z", + "git_sha": "7985ed2fed47d742c45639776b8da5e6f256cc38", + "model": "openai:gpt-5.4-mini", + "provider_route": "openrouter:openai/gpt-5.4-mini", + "runner": "tutorial-ticket-routing-experiment", + "runs": [ + { + "absolute_lift": 0.55, + "baseline_errors": 0, + "baseline_score": 0.35, + "cache": { + "bypasses": 0, + "cleared_before_run": true, + "hits": 0, + "misses": 0 + }, + "duration_ms": 9287, + "heldout_examples": 20, + "optimized_errors": 0, + "optimized_score": 0.9, + "run": 1, + "started_at": "2026-09-17T20:13:44Z", + "usage": { + "input_tokens": 14296, + "output_tokens": 466, + "requests_with_usage": 40, + "total_tokens": 14762 + } + }, + { + "absolute_lift": 0.55, + "baseline_errors": 0, + "baseline_score": 0.4, + "cache": { + "bypasses": 0, + "cleared_before_run": true, + "hits": 0, + "misses": 0 + }, + "duration_ms": 13135, + "heldout_examples": 20, + "optimized_errors": 0, + "optimized_score": 0.95, + "run": 2, + "started_at": "2026-09-17T20:13:53Z", + "usage": { + "input_tokens": 14296, + "output_tokens": 453, + "requests_with_usage": 40, + "total_tokens": 14749 + } + }, + { + "absolute_lift": 0.65, + "baseline_errors": 0, + "baseline_score": 0.3, + "cache": { + "bypasses": 0, + "cleared_before_run": true, + "hits": 0, + "misses": 0 + }, + "duration_ms": 8036, + "heldout_examples": 20, + "optimized_errors": 0, + "optimized_score": 0.95, + "run": 3, + "started_at": "2026-09-17T20:14:06Z", + "usage": { + "input_tokens": 14296, + "output_tokens": 486, + "requests_with_usage": 40, + "total_tokens": 14782 + } + } + ], + "schema_version": 2, + "scope": { + "claimed": "LabeledFewShot(k: 8) held-out lift on the shipped sixty-ticket routing task with openrouter:openai/gpt-5.4-mini across 3 live repeats", + "not_claimed": [ + "generalization beyond the shipped support-ticket dataset", + "search-optimizer (RandomSearch/MIPROv2) effectiveness", + "DSPy-matched comparison" + ] + }, + "script": "scripts/tutorial_ticket_routing_experiment.exs", + "summary": { + "all_runs_improved": true, + "baseline_score_max": 0.4, + "baseline_score_min": 0.3, + "duration_ms_max": 13135, + "duration_ms_min": 8036, + "min_absolute_lift": 0.55, + "optimized_score_max": 0.95, + "optimized_score_min": 0.9, + "optimizer_budget": { + "active_reservations": 0, + "exhausted": null, + "limits": { + "input_tokens": 1500000, + "output_tokens": 61440, + "requests": 240, + "usd": 3.0 + }, + "pricing": { + "input_per_million": 0.75, + "output_per_million": 4.5, + "source_url": "https://openai.com/api/pricing/" + }, + "requests": 120, + "reservation_ledger_version": 2, + "reservations": [], + "reserved": { + "input_tokens": 0, + "output_tokens": 0, + "usd": 0.0 + }, + "single_attempt_transport_enforced": true, + "transport_attempts": 120, + "usage": { + "input_tokens": 42888, + "output_tokens": 1405, + "usd": 0.038487999999999994 + } + }, + "repeats": 3, + "total_usage": { + "input_tokens": 42888, + "output_tokens": 1405, + "requests_with_usage": 120, + "total_tokens": 44293 + } + }, + "tutorial": "docs/TUTORIAL_TICKET_ROUTING.md" +} diff --git a/decisions.md b/decisions.md index 936b1ab0..92b4460b 100644 --- a/decisions.md +++ b/decisions.md @@ -13,7 +13,7 @@ not necessarily when it was made. | 2026-08-23 | Imp is not published to Hex; each release is a private immutable Git tag (`v0.3.x`) installed with `{:imp, github: "deepfates/imp", tag: ...}`. | `README.md` Install. `mix hex.build` refuses the Git ExMCP dependency, so `mix package.check` cannot establish Hex publishability; publication is an owner action. | Amended 2026-09-17: the repository is public, so the tags are public and need no credentials; whether to publish to Hex is still open. | ExMCP is consumable as a released Hex package and the owner chooses to publish. | | 2026-09-13 | An ACP tool kind is derived from the MCP `ToolAnnotations` the server declares, not from a host table keyed by tool name; a tool that declares no hint gets `nil` rather than a guess, and an explicit `:tool_kinds` entry still wins. | `lib/imp/acp/tool_kind.ex` moduledoc, `test/acp_options_tool_kinds_test.exs`. A name table goes stale the moment a server publishes a tool it does not name, and that is exactly when a permission mode that would have asked does not ask. | In force. | Does not retire. | | 2026-09-13 | On `session/load` and `session/resume` the session is installed with the `_meta` it was created with, not the `_meta` the request carried; both are passed to the program factory as `:meta` and `:requested_meta`. | `lib/imp/acp.ex` moduledoc, `test/acp_imp_acp_test.exs`. A session's history and transcript belong to the configuration that produced them, so a resume that silently adopts a different one replays one configuration's transcript as another's; only the factory knows which of its own `_meta` keys are identity-bearing, so it is given both and decides whether to refuse. | In force. | Does not retire. | -| 2026-09-13 | `Imp.ACP` (the ACP session/program adapter) is part of Imp; the separate `imp_acp` package is retired with no compatibility shim and no second implementation. | Retired `imp_acp` README, `AGENTS.md`. One protocol stack, one owner of the ExMCP dependency. | In force on `main`; **not in `v0.3.2`** (the released tag predates absorption). | The "unreleased" caveat retires at the next tag. The absorption itself does not retire. | +| 2026-09-13 | `Imp.ACP` (the ACP session/program adapter) is part of Imp; the separate `imp_acp` package is retired with no compatibility shim and no second implementation. | Retired `imp_acp` README, `AGENTS.md`. One protocol stack, one owner of the ExMCP dependency. | In force, and released in `v0.4.0`. | The "unreleased" caveat retired at `v0.4.0`. The absorption itself does not retire. | | 2026-09-13 | Absorption renames: `IMP_ACP_PATH` becomes `IMP_PATH` (copied workspace example), `IMP_ACP_ROOT` becomes `IMP_ROOT` (Haven preset), launcher is `examples/workspace_agent/scripts/workspace-agent-acp`. Module names `Imp.ACP`, `Imp.ACP.Host`, `Imp.ACP.MCP` and the `deepfates.com/imp-acp` wire metadata namespace keep their names. | Retired `imp_acp` README. Saved executable and environment references are updated explicitly, not by rewriting contact history. | In force. | Does not retire. | | 2026-09-13 | The workspace agent's session store keeps the `imp_acp/workspace_agent/sessions` directory name. | `examples/workspace_agent/README.md`. Preserves saved sessions; it is a storage location, not a dependency on the retired package. | In force. | Saved sessions are migrated, or the owner accepts losing restart continuity for them. | | 2026-09-13 | Ordinary Imp boot starts no protocol listener, subprocess or remote connection and does not start the ExMCP application. Protocol entry points (`Imp.ACP.*`, non-empty `Imp.MCP.connect/2`) start it explicitly. Releases using them declare `applications: [ex_mcp: :load]`. | `mix.exs` dependency comment, `docs/PRODUCTION_OPERATIONS.md` "Protocol runtime in releases", commit `728f8c77`. Prediction and optimizer processes must not open listeners or acquire protocol boot output. | In force. | Does not retire while Imp is a library inside a host application. | diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 3875d0fe..b56e61d6 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -44,7 +44,7 @@ Also accepts `OPENROUTER_API_KEY` with `openai/gpt-5.4-mini`. Set - **Needs:** an OpenAI or OpenRouter key. No Python, no dataset download; the sixty tickets ship in the package at `priv/tutorial/support_tickets.json`. -- **Time:** 8–9 seconds per repeat, three repeats by default. +- **Time:** 8–13 seconds per repeat, three repeats by default. - **Cost:** about `$0.013` per repeat at `gpt-5.4-mini` prices; the script enforces a hard budget of `$1.00` per repeat and refuses to exceed it. - **What it measures:** held-out accuracy of a four-way enum router before and diff --git a/docs/EVIDENCE.md b/docs/EVIDENCE.md index 7fa17b58..735d06dd 100644 --- a/docs/EVIDENCE.md +++ b/docs/EVIDENCE.md @@ -45,9 +45,9 @@ compares and what it deliberately does not, are in Two held-out results, both small, both in RESULTS.md. Rows R1 and R2 are the ticket-routing tutorial: a zero-shot enum router scored -0.30–0.50 on twenty held-out tickets and 0.95–1.00 after `LabeledFewShot(k: 8)`, -over three live repeats at about a cent each. A stranger with an API key can run -that command and get their own numbers. It is the only end-to-end effectiveness +0.30–0.50 on twenty held-out tickets and 0.90–1.00 after `LabeledFewShot(k: 8)`, +over three live repeats at about a cent each, in each of two runs a month apart. +A stranger with an API key can run that command and get their own numbers. It is the only end-to-end effectiveness claim here that is reproducible from scratch. Row R6 is `examples/deployment/agent_optimization.exs`, the first ordinary Imp diff --git a/docs/IMP_FOR_DSPY_USERS.md b/docs/IMP_FOR_DSPY_USERS.md index 2c0259b3..bc7e7f8f 100644 --- a/docs/IMP_FOR_DSPY_USERS.md +++ b/docs/IMP_FOR_DSPY_USERS.md @@ -76,7 +76,7 @@ own held-out data. DSPy also has a larger Python integration ecosystem. Imp provides extension boundaries through `Imp.LM`, `Imp.Retrieve`, adapters, tools, and trainer clients; integrations written for Python do not automatically work on the -BEAM. DSPy's Flex code optimizer is not included in `0.3.2`. +BEAM. DSPy's Flex code optimizer is not included in `0.4.0`. ## Nearby Elixir work @@ -90,7 +90,7 @@ lineage to a broad programming and optimization model built for OTP deployment. ## Coming from DSPy: the five-minute version ```elixir -# pip install dspy → {:imp, github: "deepfates/imp", tag: "v0.3.2"} +# pip install dspy → {:imp, github: "deepfates/imp", tag: "v0.4.0"} # dspy.configure(lm=lm) → lm = Imp.req_llm("openai:gpt-5.4-mini", api_key: ...) # dspy.Predict("q -> a") → program = Imp.predict("q -> a", lm: lm) # program(q="...") → {:ok, pred} = Imp.call(program, %{q: "..."}) diff --git a/docs/TUTORIAL_TICKET_ROUTING.md b/docs/TUTORIAL_TICKET_ROUTING.md index 9268c3ab..219e76e9 100644 --- a/docs/TUTORIAL_TICKET_ROUTING.md +++ b/docs/TUTORIAL_TICKET_ROUTING.md @@ -4,10 +4,11 @@ Let's take the support-ticket router from the [README](../README.md) and do what you cannot do with a prompt string: score it on held-out data, improve it with an optimizer, and prove the improvement on tickets it has never seen. -The zero-shot router scored **30–50%** on twenty held-out tickets; the -optimized router scored **95–100%** on the same twenty in all three repeats. -Each full experiment — baseline, optimization, and held-out evaluation — cost -about **$0.013** and ran in **8–9 seconds** with `gpt-5.4-mini`. +Across two runs a month apart, the zero-shot router scored **30–50%** on twenty +held-out tickets; the optimized router scored **90–100%** on the same twenty in +every repeat of both. Each full experiment — baseline, optimization, and +held-out evaluation — used about 14,800 tokens and ran in **8–13 seconds** with +`gpt-5.4-mini`, for an estimated **$0.013** at list prices. Those are rows R1 and R2 in [benchmarks/RESULTS.md](https://github.com/deepfates/imp/blob/main/benchmarks/RESULTS.md), @@ -105,7 +106,8 @@ the misses are not random — they are the model guessing what squad names mean: It reads the tickets fine. It cannot know that atlas is the money squad. On a task this small the exact score moves a little between runs — our current -repeats landed between 0.30 and 0.50 — but every run tells the same story. +repeats landed between 0.30 and 0.50 across both runs — but every run tells the +same story. ## Improve With Measured Lift @@ -139,12 +141,14 @@ optimized = Imp.evaluate(compiled, testset, metric, max_concurrency: 8, timeout: #=> {0.3, 0.95} ``` -The three repeats measured 50% → 95%, 35% → 100%, and 30% → 95%: gains of -45–65 points on held-out tickets. Each run used about 14,800 tokens, cost about -$0.013, and finished in 8–9 seconds; all 120 evaluation calls completed without -a row error and the cache was cleared before each repeat, so every call was -live. Twenty rows move in 5-point steps, so trust the direction and the -magnitude, not the endpoints. Rows R1 and R2 in +The most recent three repeats measured 35% → 90%, 40% → 95%, and 30% → 95%: +gains of 55–65 points on held-out tickets. Each run used about 14,800 tokens and +finished in 8–13 seconds, an estimated $0.013 at list prices; all 120 evaluation +calls completed without a row error and the cache was cleared before each +repeat, so every call was live. An earlier run of the same command reached +95–100% optimized with gains of 45–65 points, so treat 90% as the low end you +should expect, not a regression. Twenty rows move in 5-point steps, so trust the +direction and the magnitude, not the endpoints. Rows R1 and R2 in [benchmarks/RESULTS.md](https://github.com/deepfates/imp/blob/main/benchmarks/RESULTS.md). It is not a magic button. The remaining misses are genuinely marginal tickets diff --git a/examples/deployment/README.md b/examples/deployment/README.md index 2e46fc5e..e85dd44e 100644 --- a/examples/deployment/README.md +++ b/examples/deployment/README.md @@ -24,7 +24,7 @@ IMP_PATH=../.. mix deps.get IMP_PATH=../.. mix run --no-start run_workflow.exs ``` -A copied application can omit `IMP_PATH` and resolve the immutable `v0.3.2` +A copied application can omit `IMP_PATH` and resolve the immutable `v0.4.0` Git dependency declared in `mix.exs`. The workflow: diff --git a/examples/deployment/mix.exs b/examples/deployment/mix.exs index 7a7d03a6..71fd8734 100644 --- a/examples/deployment/mix.exs +++ b/examples/deployment/mix.exs @@ -11,7 +11,7 @@ defmodule ImpDeployment.MixProject do defp deps do case System.get_env("IMP_PATH") do - nil -> [{:imp, github: "deepfates/imp", tag: "v0.3.2"}] + nil -> [{:imp, github: "deepfates/imp", tag: "v0.4.0"}] path -> [{:imp, path: path}] end end diff --git a/livebooks/01_real_lm_front_door.livemd b/livebooks/01_real_lm_front_door.livemd index 67104c05..05f96d42 100644 --- a/livebooks/01_real_lm_front_door.livemd +++ b/livebooks/01_real_lm_front_door.livemd @@ -30,7 +30,7 @@ if repo do Mix.install([{:imp, path: repo}], install_opts) else # Standalone notebook: install the tagged Git source release. - Mix.install([{:imp, github: "deepfates/imp", tag: "v0.3.2"}]) + Mix.install([{:imp, github: "deepfates/imp", tag: "v0.4.0"}]) end api_key = System.get_env("OPENAI_API_KEY") || System.get_env("LB_OPENAI_API_KEY") diff --git a/livebooks/02_programming_not_prompting.livemd b/livebooks/02_programming_not_prompting.livemd index e3ce8960..f1cc9442 100644 --- a/livebooks/02_programming_not_prompting.livemd +++ b/livebooks/02_programming_not_prompting.livemd @@ -30,7 +30,7 @@ if repo do Mix.install([{:imp, path: repo}], install_opts) else # Standalone notebook: install the tagged Git source release. - Mix.install([{:imp, github: "deepfates/imp", tag: "v0.3.2"}]) + Mix.install([{:imp, github: "deepfates/imp", tag: "v0.4.0"}]) end live_provider_enabled? = System.get_env("LIVE_PROVIDER") == "1" ``` diff --git a/livebooks/03_evaluate_and_optimize.livemd b/livebooks/03_evaluate_and_optimize.livemd index c6b015b9..e3395ec0 100644 --- a/livebooks/03_evaluate_and_optimize.livemd +++ b/livebooks/03_evaluate_and_optimize.livemd @@ -30,7 +30,7 @@ if repo do Mix.install([{:imp, path: repo}], install_opts) else # Standalone notebook: install the tagged Git source release. - Mix.install([{:imp, github: "deepfates/imp", tag: "v0.3.2"}]) + Mix.install([{:imp, github: "deepfates/imp", tag: "v0.4.0"}]) end live_provider_enabled? = System.get_env("LIVE_PROVIDER") == "1" ``` diff --git a/livebooks/04_tools_agents_mcp_rlm.livemd b/livebooks/04_tools_agents_mcp_rlm.livemd index 1414bcec..401bc5f2 100644 --- a/livebooks/04_tools_agents_mcp_rlm.livemd +++ b/livebooks/04_tools_agents_mcp_rlm.livemd @@ -30,7 +30,7 @@ if repo do Mix.install([{:imp, path: repo}], install_opts) else # Standalone notebook: install the tagged Git source release. - Mix.install([{:imp, github: "deepfates/imp", tag: "v0.3.2"}]) + Mix.install([{:imp, github: "deepfates/imp", tag: "v0.4.0"}]) end live_provider_enabled? = System.get_env("LIVE_PROVIDER") == "1" ``` diff --git a/livebooks/05_operate_and_live_checks.livemd b/livebooks/05_operate_and_live_checks.livemd index 7c47384c..7b6d81af 100644 --- a/livebooks/05_operate_and_live_checks.livemd +++ b/livebooks/05_operate_and_live_checks.livemd @@ -30,7 +30,7 @@ if repo do Mix.install([{:imp, path: repo}], install_opts) else # Standalone notebook: install the tagged Git source release. - Mix.install([{:imp, github: "deepfates/imp", tag: "v0.3.2"}]) + Mix.install([{:imp, github: "deepfates/imp", tag: "v0.4.0"}]) end live_provider_enabled? = System.get_env("LIVE_PROVIDER") == "1" ``` diff --git a/mix.exs b/mix.exs index a5484256..a1bef226 100644 --- a/mix.exs +++ b/mix.exs @@ -4,7 +4,7 @@ defmodule Imp.MixProject do def project do [ app: :imp, - version: "0.3.2", + version: "0.4.0", elixir: "~> 1.19", name: "Imp", source_url: "https://github.com/deepfates/imp", diff --git a/priv/public_api.json b/priv/public_api.json index 3c575bbf..4cd98b42 100644 --- a/priv/public_api.json +++ b/priv/public_api.json @@ -9699,7 +9699,7 @@ "types": [] } ], - "package_version": "0.3.2", + "package_version": "0.4.0", "schema_version": 3, "scope": "Curated Imp API compiled from package-shipped sources. Exports come from non-hidden documentation entries; callbacks, types, and struct fields are category- and policy-gated." } diff --git a/scripts/tutorial_ticket_routing_experiment.exs b/scripts/tutorial_ticket_routing_experiment.exs index adf73dbc..99ceb6f9 100644 --- a/scripts/tutorial_ticket_routing_experiment.exs +++ b/scripts/tutorial_ticket_routing_experiment.exs @@ -159,9 +159,9 @@ defmodule TutorialTicketRoutingExperiment do "doc_claims_under_test" => %{ "source" => "docs/TUTORIAL_TICKET_ROUTING.md", "baseline_repeat_range" => [0.30, 0.50], - "optimized_repeat_range" => [0.95, 1.0], + "optimized_repeat_range" => [0.90, 1.0], "cost_usd_per_repeat_approx" => 0.013, - "duration_seconds_range" => [8, 9] + "duration_seconds_range" => [8, 13] }, "scope" => %{ "claimed" => diff --git a/test/deployment_reference_test.exs b/test/deployment_reference_test.exs index 4488ee0c..9a2486a7 100644 --- a/test/deployment_reference_test.exs +++ b/test/deployment_reference_test.exs @@ -454,7 +454,7 @@ defmodule DeploymentReferenceTest do readme = File.read!(Path.join(@example_root, "README.md")) assert mix_file =~ ~s(elixir: "~> 1.19") - assert mix_file =~ ~s({:imp, github: "deepfates/imp", tag: "v0.3.2"}) + assert mix_file =~ ~s({:imp, github: "deepfates/imp", tag: "v0.4.0"}) assert mix_file =~ "IMP_PATH" assert readme =~ "bounded supervised task" assert readme =~ "IMP_MODEL" diff --git a/test/package_contract_test.exs b/test/package_contract_test.exs index c67759a2..667a131b 100644 --- a/test/package_contract_test.exs +++ b/test/package_contract_test.exs @@ -106,11 +106,11 @@ defmodule PackageContractTest do assert_release_files(files) end - test "the 0.3.2 release surfaces agree" do + test "the 0.4.0 release surfaces agree" do version = Mix.Project.config()[:version] source_dependency = ~s({:imp, github: "deepfates/imp", tag: "v#{version}"}) - assert version == "0.3.2" + assert version == "0.4.0" assert File.read!("RELEASE_NOTES.md") =~ "# Imp v#{version}" assert File.read!("CHANGELOG.md") =~ "## #{version}" assert File.read!("examples/deployment/mix.exs") =~ source_dependency