Skip to content

Fix: the embedding model default breaks one deployment or the other - #174

Merged
adamjohnwright merged 2 commits into
mainfrom
fix/embedding-model-default
Sep 4, 2026
Merged

Fix: the embedding model default breaks one deployment or the other#174
adamjohnwright merged 2 commits into
mainfrom
fix/embedding-model-default

Conversation

@adamjohnwright

@adamjohnwright adamjohnwright commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Current main fails on the first query for any Reactome deployment that does not set EMBEDDING_MODEL.

What is actually wrong

graph.py defaulted to the literal "bge-m3". Against api.openai.com:

The model `bge-m3` does not exist or you do not have access to it. (404)

But bge-m3 is not a bad model. It is the correct one for Plant Reactome, which serves it from a self-hosted OpenAI-compatible endpoint via OPENAI_BASE_URL — the bundle is openai/bge-m3/plantreactome/Release68. It is wrong only as a default, because every bundle published for Reactome uses text-embedding-3-large.

So hardcoding either model breaks the other deployment. My first attempt at this fix hardcoded the Reactome bundle and would have broken Plant Reactome; this corrects that.

The fix

The bundle path records the model that built it, which makes it the source of truth rather than a constant kept in sync by hand. resolve_embedding_model():

  1. reads the model from all installed bundles
  2. reports it as a misconfiguration when they disagree — AgentGraph builds one embedding shared by every profile, so bundles built with different models cannot all be served
  3. honours EMBEDDING_MODEL as an override, and logs an error when the override disagrees with the bundle, since that gives meaningless retrieval rather than an obvious failure
  4. falls back to text-embedding-3-large only when nothing is installed

Verified against the real Release95 bundle: resolves to text-embedding-3-large.

Where it came from, and a process point

The default arrived in a8aa04c"Initial commit of adding PlantReactome profile", Justin Elser, 2026-08-19.

That branch has no pull request, in any state. It reached main today through the integration PR #167 without ever having had a review of its own, and it carried a default that breaks the primary deployment. Neither the lint nor the type gate can see this: it is a runtime configuration mismatch, and nothing exercised it.

Plant Reactome is unaffected in every configuration, verified:

bundle installed EMBEDDING_MODEL resolves to
openai/bge-m3/plantreactome/Release68 unset bge-m3
same bge-m3 bge-m3
same something else the override, with a loud error

It is in fact more robust than before: it previously relied on a hardcoded default that happened to match, so changing that default for Reactome would have silently broken Plant Reactome. It now derives from the bundle actually installed.

Also

EmbeddingEnvironment.get_model raises KeyError for an uninstalled database while its sibling get_dir returns None. Not changed here — it is a shared API — but worth knowing.

Tests

Seven: Reactome resolves to text-embedding-3-large, Plant Reactome to bge-m3, override honoured, override-disagrees reported, mixed bundles reported, nothing-installed falls back, and one asserting the default is specifically never bge-m3 when a Reactome bundle is installed.

Deployed instances

beta.reactome.org/chat and production both run image e398a37, which predates this code, so neither is affected.

🤖 Generated with Claude Code

adamjohnwright and others added 2 commits September 4, 2026 21:28
EMBEDDING_MODEL defaulted to the literal "bge-m3", which OpenAI has no such
model for, so any deployment that did not set the variable got a 404 on its
first query:

  The model `bge-m3` does not exist or you do not have access to it.

Confirmed against the API. This arrived on main with the plantreactome merge
earlier today; the lint and type gates cannot see it because it is a runtime
configuration mismatch, and nothing exercised it.

Queries are embedded and compared against vectors already in Chroma, so the
model that built the bundle is the only correct answer -- not a constant kept in
sync by hand. resolve_embedding_model() now reads it from the installed bundle,
falls back to text-embedding-3-large when nothing is installed, and still honours
EMBEDDING_MODEL. A configured value that disagrees with the bundle is logged as
an error, because that combination produces meaningless retrieval rather than an
obvious failure.

Note EmbeddingEnvironment.get_model raises KeyError for a database that is not
installed, while its sibling get_dir returns None for the same condition. Handled
at the call site rather than changed, since it is a shared API.

Five tests, including one asserting the default is never "bge-m3" specifically.
env_template and the beta template now document both model variables.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ctome's

Corrects the previous commit, which hardcoded the Reactome bundle and so would
have broken Plant Reactome instead.

"bge-m3" is not a bad model -- it is the right one for Plant Reactome, which
serves it from a self-hosted OpenAI-compatible endpoint via OPENAI_BASE_URL. It
is wrong only as a *default*, because every bundle published for Reactome uses
text-embedding-3-large and gets a 404 from api.openai.com. Hardcoding either
model breaks the other deployment, so neither is hardcoded: the bundle path
records which model built it, and that is the source of truth.

AgentGraph builds one embedding shared by every profile, so all installed
bundles must agree. Bundles built with different models are now reported as the
misconfiguration they are, rather than one of them silently returning nonsense.

The default arrived in a8aa04c, "Initial commit of adding PlantReactome
profile", which reached main through the integration PR without ever having had
a pull request of its own.

Seven tests: Reactome resolves to text-embedding-3-large, Plant Reactome to
bge-m3, an override is honoured, an override disagreeing with the bundle is
reported, mixed bundles are reported, and nothing installed falls back.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@adamjohnwright adamjohnwright changed the title Fix: EMBEDDING_MODEL default is a model OpenAI does not have Fix: the embedding model default breaks one deployment or the other Sep 4, 2026
@adamjohnwright
adamjohnwright merged commit 0ae1c8b into main Sep 4, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant