BM25 Wins at Scale, Not Everywhere

BM25 is not a universal winner. A July 2026 arXiv study compared lexical, dense, graph-based, and agentic retrieval across 28 nested enterprise-corpus tiers, from 1,144 to 511,959 documents, while holding the questions, reader, and judging protocol fixed. File-System Agent led the smallest tiers; BM25 overtook it around 10 million corpus tokens and led at every larger shared tier. 1

The product problem is choosing a retriever from one leaderboard. In the paper's full corpus, BM25 scored 50.5 versus 30.7 for File-System Agent and 29.9 for DenseRAG; the agent also used 226K query tokens at the smallest tier versus BM25's 5.8K. Treat those as study results, not deployment guarantees: the benchmark is a fictional enterprise corpus with 500 questions, one reader model, and incomplete graph tiers. 1

First build: create nested versions of your own corpus, keep the reader, prompts, questions, and budget unchanged, then A/B BM25, dense retrieval, and your current agent path. Track answer quality, evidence recall, p95 latency, and query cost together. Use the result as a reversible routing rule: ranked retrieval first, agentic reasoning after discovery when the question needs aggregation.