Not a blocker for the LangChain upgrade or the retriever rewrite — filing so the tooling is not lost.
What exists
bin/retrieval_baseline and tests/golden/questions.txt were written during the Sept 4 tooling work. The harness:
- feeds a fixed 20-question set directly to each retriever, bypassing the multi-query expansion, so BM25 and plain vector search are exactly reproducible (the expansion and SelfQuery both call an LLM and are not)
- records documents by Reactome stable ID (
st_id), which survives a bundle rebuild
capture writes JSON; compare diffs two captures; overlap measures how far plain vector search lands from SelfQuery within one capture
Why it is NOT being used as a regression gate now
The current fusion has defects, so pinning current behaviour would make fixing them look like breaking the test:
- BM25 and vector results are concatenated, not fused —
doc_lists.append(bm25_docs + vector_docs). RRF then runs across query variants, not across retrievers. Despite the name, this is not hybrid fusion.
- Because BM25 returns
k=10 first, every vector result enters RRF at rank 11+, so a vector top hit scores 1/71 against BM25 1/61.
- Ties are broken by list position (
sorted is stable), so BM25 and earlier query variants win ties.
- The
[1/n]*n weights are uniform, so the weighting cannot affect ordering at all.
The conversation flow is also expected to change substantially — running analysis on user data through the chat is being explored — so a baseline captured against todays flow would age out quickly.
What is still worth running now
overlap answers a live design question with data rather than judgement: how much does plain semantic search agree with SelfQueryRetriever on the same question? That informs replacing SelfQuery with a plain semantic retriever (keeping BM25), which would remove ~20 of the ~21 LLM calls per message and the component most likely to break on LangChain 1.x.
Later
Once the retriever settles, capture a baseline against the fixed fusion and use compare as a gate from then on. The question set is meant to be edited — it is a fixture, not a contract.
🤖 Generated with Claude Code
Not a blocker for the LangChain upgrade or the retriever rewrite — filing so the tooling is not lost.
What exists
bin/retrieval_baselineandtests/golden/questions.txtwere written during the Sept 4 tooling work. The harness:st_id), which survives a bundle rebuildcapturewrites JSON;comparediffs two captures;overlapmeasures how far plain vector search lands from SelfQuery within one captureWhy it is NOT being used as a regression gate now
The current fusion has defects, so pinning current behaviour would make fixing them look like breaking the test:
doc_lists.append(bm25_docs + vector_docs). RRF then runs across query variants, not across retrievers. Despite the name, this is not hybrid fusion.k=10first, every vector result enters RRF at rank 11+, so a vector top hit scores1/71against BM251/61.sortedis stable), so BM25 and earlier query variants win ties.[1/n]*nweights are uniform, so the weighting cannot affect ordering at all.The conversation flow is also expected to change substantially — running analysis on user data through the chat is being explored — so a baseline captured against todays flow would age out quickly.
What is still worth running now
overlapanswers a live design question with data rather than judgement: how much does plain semantic search agree withSelfQueryRetrieveron the same question? That informs replacing SelfQuery with a plain semantic retriever (keeping BM25), which would remove ~20 of the ~21 LLM calls per message and the component most likely to break on LangChain 1.x.Later
Once the retriever settles, capture a baseline against the fixed fusion and use
compareas a gate from then on. The question set is meant to be edited — it is a fixture, not a contract.🤖 Generated with Claude Code