feat: Add typesafe.ai as provider and automation - #1267
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e9c280075d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| "window", | ||
| "js", | ||
| "converse", | ||
| "typesafe", |
There was a problem hiding this comment.
Adding typesafe as a format here implements provider behavior in the deprecated proxy package, so the active gateway will not receive the corresponding routing, translation, and secret handling. Keep the model_list.json catalog entries here, but move the schema, provider, and automation integration to the parent repository's gateway/ directory as required.
AGENTS.md reference: AGENTS.md:L10-L15
Useful? React with 👍 / 👎.
Erin McNulty (erin2722)
left a comment
Erin McNulty (erin2722)
left a comment
There was a problem hiding this comment.
thanks! we will have to add gateway support for this before we can advertise the models as available in our model list, Alex Z (@CLowbrow) is adding a temporary/hacky solution to allow typesafe for scorers today and then after that I think we can merge this in tandem with gateway PRs that add support.
|
Erin McNulty (@erin2722) my main objective here was having price data on typesafe.ai spans created through the SDK. Do we really need to add gateway support before merging this? I wasn't expecting the systems to be so interconnected while unrelated. |
|
ahhhh I see now, I actually didn't know that the SDK pulls prices from here as well, but I suppose that makes sense. yeah there is some really annoying intermingling here, let me just get a main repo PR up that will make sure we don't accidentally expose typesafe as a provider before we support it |
|
Erin McNulty (@erin2722) Amazing ty! The SDK doesn't pull prices directly but the backend does based on the provider and model the SDK sends in the metadata. |
|
ok we can merge this and then will just update the main repo with https://github.com/braintrustdata/braintrust/pull/20608 so that this doesn't show up in our model pickers and whatnot before it is supported by Alex Z (@CLowbrow) 's work |
## Summary The shared model catalog (`proxy` `model_list.json`) now includes TypeSafe `jev-*` models with `"flavor": "evaluation"`, added in braintrustdata/braintrust-proxy#1267. The Rust `ModelFlavor` enum only knew `chat` / `completion` / `embedding` / `responses`, so deserializing the catalog fails with `unknown variant "evaluation"`, panicking gateway/api-rs startup. This adds the `Evaluation` variant so the shared catalog deserializes. ## Notes - `#[serde(rename_all = "lowercase")]` maps the variant to `"evaluation"`. - No exhaustive `match` on `ModelFlavor` exists in the tree, so this is purely additive. ## Validation - `cargo test -p braintrust-llm-router` passes. - In the consuming `braintrust` repo (with the proxy submodule bumped to include #1267): the previously-panicking gateway tests now pass — `cargo test -p gateway --lib proxy::router_builder::tests` (36 passed) and `server::tests::supports_modern_and_legacy_prefixes` — and `cargo fmt -- --check` is clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Starfolk <noreply@starfolk.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2 participants
https://docs.typesafe.ai/models