Retrieval: duplicate documents consume half the top-k budget on reactions

[adamjohnwright]

Measured with bin/retrieval_baseline against openai/text-embedding-3-large/reactome/Release95, 20 questions, k=10.

The number

Distinct documents actually returned, out of k=10:

collection	retriever	distinct	worst case
reactions	vector	4.8	4
reactions	selfquery	4.8	4
complexes	vector	7.2	3
complexes	selfquery	6.6	3
ewas / summations	all	~10.0	9–10
all collections	bm25	10.0	10

Vector search on reactions returns ten results containing about five distinct reactions. For "How does TP53 regulate PTEN transcription?" the top five were two reactions repeated.

Cause

reactions.csv has one row per reaction per pathway / input / output / catalyst combination, so a reaction in several pathways becomes several rows. Those rows have different page_content, and weighted_reciprocal_rank de-duplicates on page_content, so they survive fusion as separate documents.

BM25 is unaffected — its scoring spreads across distinct text — which is why only the vector side shows it.

Effect

The LLM receives a context window roughly half-filled with restatements of the same reaction, and the effective k on the largest collection is about 5 while paying the cost of 10.

Options

• de-duplicate on st_id before fusion (RRF already accepts an id_key)
• over-fetch and collapse to k distinct stable IDs
• change the bundle so one row is one reaction, with pathways/inputs as list metadata — a data-generation change, more invasive

Independent of the LangChain version, so it can be fixed before or after the upgrade.

Reproduce:

./bin/retrieval_baseline capture --out before.json --with-selfquery

🤖 Generated with Claude Code

[adamjohnwright]

Fixed in 2c98316 on integrate/tooling-and-branch-merges (PR #167).

Verified against openai/text-embedding-3-large/reactome/Release95:

reactions    4 -> 10 distinct of 10   How does TP53 regulate PTEN transcription?
reactions    5 -> 10 distinct of 10   Describe the components of the proteasome
reactions    5 -> 10 distinct of 10   What are the steps of glycolysis?
complexes    9 -> 10 distinct of 10   How does TP53 regulate PTEN transcription?
complexes    7 -> 10 distinct of 10   Describe the components of the proteasome

Vector search now over-fetches 3x and collapses to the highest-ranked row per st_id; BM25 and vector results are appended as separate lists so both top hits score 1/61. Four tests added covering de-duplication order, the limit, the no-st_id fallback, and equal scoring of the two retrievers' top hits.

🤖 Generated with Claude Code