Retrieval: replace SelfQueryRetriever with a plain semantic retriever?

[adamjohnwright]

Proposed by Helia: drop SelfQueryRetriever, keep BM25 plus vector search. This records the evidence, because the measurement did not come out the way that would make it a free change.

The case for dropping it

• Cost. SelfQueryRetriever makes one LLM call per collection per query variant to translate the question into a structured query. With 4 collections and 5 query variants (4 generated + the original) that is 20 LLM calls per user message, plus 1 for the expansion. A plain semantic retriever has no LLM in the loop.
• Upgrade risk. It depends on lark and on structured-output behaviour that has moved across LangChain versions. It is the component most likely to break on 1.x, and it is one of five places HybridRetriever reaches into LangChain internals.

The case against assuming it is free

Measured with bin/retrieval_baseline, 20 questions, k=10, Release95.

Overlap between SelfQuery and plain vector search:

collection	mean overlap	identical results
complexes	0.33	0/20
ewas	0.46	3/20
reactions	0.60	4/20
summations	0.54	1/20
overall	0.48

Run-to-run stability, same code, same questions, two runs:

retriever	self-overlap	byte-identical
bm25	1.00	80/80
vector	0.99	78/80
selfquery	1.00	78/80

SelfQuery is stable, not noisy. Its ~half-different results are real and repeatable, so removing it is a genuine retrieval change rather than dropping a source of churn. Note it does not only build a metadata filter — it also rewrites the query string before searching, which is likely part of the divergence.

What the difference looks like

"How does TP53 regulate PTEN transcription?" — top hit:

• selfquery: PTEN gene transcription is stimulated by TP53
• vector: TP53 binds the PTEN promoter

Both defensible. Set overlap cannot say which is better; that needs a quality eval (the ragas work), not a diff.

Suggested order

Settle #169 (duplicates consume half the top-k on reactions) and #170 (BM25 and vector are concatenated, not fused) first. Both change what the vector side returns, so measuring SelfQuery against a fixed baseline is more meaningful afterwards.

Reproduce:

./bin/retrieval_baseline capture --out c.json --with-selfquery
./bin/retrieval_baseline overlap c.json

🤖 Generated with Claude Code

[adamjohnwright]

Corrected numbers

The overlap metric in the original post used set(), which collapses duplicates — exactly the duplication issue #169 is about. On reactions it was comparing ~5 distinct documents rather than 10 results, so it measured less than it appeared to. bin/retrieval_baseline overlap now reports three columns:

collection	set overlap	rank agreement	distinct/k
complexes	0.33	0.11	0.72
ewas	0.46	0.15	1.00
reactions	0.60	0.30	0.47
summations	0.54	0.20	1.00
overall	0.48	0.19	0.80

Rank agreement is 0.19 — SelfQuery and plain vector search put the same document in the same position only about a fifth of the time. The divergence is larger than the set-overlap figure suggested, not smaller.

That does not change the direction of the decision, but it does raise the stakes: replacing SelfQuery with a plain semantic retriever is a substantial change to what reaches the LLM, and the ordering changes more than the membership does.

distinct/k restates #169 from this angle: on reactions, 47% of the vector result is non-repeated, so roughly half the budget goes to duplicates.

🤖 Generated with Claude Code