feat(api): discover a provider's models from the provider itself - #652
Draft
jayesh9747 wants to merge 1 commit into
Draft
feat(api): discover a provider's models from the provider itself#652jayesh9747 wants to merge 1 commit into
jayesh9747 wants to merge 1 commit into
Conversation
The shipped catalog is a hand-maintained preset list, so a well-known
provider offers only the models checked into model-catalog.yaml —
google-gemini ships 2 against openai's 5 and anthropic's 6 — and it goes
stale whenever a provider releases a model.
The catalog is also the only source of model properties. A model
configured without them has no context_length, and
resolveCompactionThresholdTokens then falls back to
DEFAULT_CONTEXT_COMPACTION_THRESHOLD_TOKENS (50k) instead of 80% of the
real window. For a 1M-context Gemini model that compacts roughly 17x
earlier than it should, silently.
Add GET /settings/model-providers/{name}/discovered-models, which asks
the provider which models it serves using the stored key. Gemini's
native list reports inputTokenLimit and outputTokenLimit, so discovered
Gemini models carry the properties compaction needs; the remaining
providers use the OpenAI-compatible list, which reports ids only, and
those models are returned without invented limits.
The key is read from the stored manifest and never crosses the wire.
Gemini takes it as a query parameter and the compatible shape as a
bearer header; both are covered by tests.
Refs truefoundry#648
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 62d7821 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes part of #648.
Problem
A well-known provider can only offer the models checked into
packages/trueforge/catalog/model-catalog.yaml.google-geminiships 2 there, againstopenai's 5 andanthropic's 6, and the file goes stale whenever a provider releases a model. Google's own list endpoint currently reports 40 chat-capable models.The catalog is also the only source of model properties, which turns a stale list into a silent runtime problem. A model configured without them has no
context_length, soresolveCompactionThresholdTokensfalls back toDEFAULT_CONTEXT_COMPACTION_THRESHOLD_TOKENS(50k) rather than 80% of the real window:That is roughly 17x earlier than it should, with nothing surfaced to the user.
Change
Adds
GET /api/v1/settings/model-providers/{name}/discovered-models, which asks a configured provider which models it actually serves, using the stored API key.inputTokenLimit/outputTokenLimit— so discovered Gemini models arrive with the properties compaction needs.GET /models, which reports ids only. Those models are returned without properties rather than with invented ones.truefoundryresolves its endpoint and token at runtime, so it reports unsupported (501).The API key is read from the stored manifest and never crosses the wire. Gemini takes it as a query parameter, the compatible shape as a bearer header; both placements are asserted in tests.
Responses:
200/404unknown provider /501no adapter /502provider unreachable or rejecting.Verified
Against a local
docker composestack with a real Gemini key:gemini-3.8-flash→context_length: 1048576,max_output_tokens: 65536Notes for review
ModelCatalogServer, which is defined in@truefoundry/assistant-ui-runtime(external dependency, pinned 0.1.27) and has nodiscoverModels. Wiring the UI needs an optional method added there first — the same shape as the existingdeleteModelProvider?. This PR is the backend half so it can land independently.pnpm sdk:generate, since CI cannot commit those to a fork PR.pnpm-lock.yamlis unchanged.