Retrieval: BM25 and vector results are concatenated, not fused

[adamjohnwright]

In HybridRetriever.retrieve_documents (src/retrievers/csv_chroma.py):

for i, query in enumerate(queries):
    bm25_docs = bm25_retriever.invoke(query, ...)
    vector_docs = vector_retriever.invoke(query, ...)
    doc_lists.append(bm25_docs + vector_docs)   # concatenated into ONE list
subdirectory_docs.extend(self.weighted_reciprocal_rank(doc_lists))

Each per-query list is BM25s k results followed by the vector retrievers. RRF then runs across query variants, not across retrievers. Despite the name, BM25 and vector search are never fused against each other.

Three consequences follow from that one line.

1. Vector results are systematically down-ranked

RRF scores by position in the list (weight / (rank + 60), rank from 1). With BM25 returning k=10 first, every vector result enters at rank 11+:

• BM25 top hit: 1/61
• vector top hit: 1/71

About 14% — modest, because c=60 flattens the curve — but it falls out of iteration order rather than from any decision.

2. Ties are broken by list position

Documents with equal RRF scores are ordered by first appearance in chain.from_iterable(doc_lists), because sorted is stable. So on a tie, earlier query variants win, and BM25 beats the vector store. Deterministic, but arbitrary. Pinned by tests/retrievers/test_hybrid_retriever.py::test_ties_are_broken_by_position_not_by_score.

3. The weights are inert

create_bm25_chroma_ensemble_retriever passes [1 / len(doc_lists)] * len(doc_lists). A constant multiplier across every list cannot change ordering — only absolute scores. The weighting currently does nothing. Pinned by test_weights_are_uniform_so_they_cannot_change_the_ordering.

Fix

Append BM25 and vector results as separate lists:

doc_lists.append(bm25_docs)
doc_lists.append(vector_docs)

Both top hits then score 1/61, and the weights become a real BM25-vs-vector dial — which is presumably what "weighted" was for.

This is a retrieval-quality change, not a refactor. Worth measuring with bin/retrieval_baseline before and after.

Related: #169 (duplicates eat the top-k), #168 (baseline harness).

🤖 Generated with Claude Code

[adamjohnwright]

Fixed in 2c98316 on integrate/tooling-and-branch-merges (PR #167).

Verified against openai/text-embedding-3-large/reactome/Release95:

reactions    4 -> 10 distinct of 10   How does TP53 regulate PTEN transcription?
reactions    5 -> 10 distinct of 10   Describe the components of the proteasome
reactions    5 -> 10 distinct of 10   What are the steps of glycolysis?
complexes    9 -> 10 distinct of 10   How does TP53 regulate PTEN transcription?
complexes    7 -> 10 distinct of 10   Describe the components of the proteasome

Vector search now over-fetches 3x and collapses to the highest-ranked row per st_id; BM25 and vector results are appended as separate lists so both top hits score 1/61. Four tests added covering de-duplication order, the limit, the no-st_id fallback, and equal scoring of the two retrievers' top hits.

🤖 Generated with Claude Code