LLMs can retrieve. Embeddings still win the default path.

Replacing an embedding index with an LLM sounds attractive when retrieval needs reasoning. The Embedder's Dilemma compares 10 LLMs with 26 embedding models across 37 MTEB(LLM) tasks. The best scores are nearly tied: 77.6 for Gemini 3.1 Pro and 77.2 for Octen-8B. LLMs lead retrieval by 8.5 points; embeddings lead classification by 5.6 points. 1

That split changes the product architecture. One benchmark pass cost $154.14 for Gemini 3.1 Pro and $0.108 for Octen-8B. The paper also reports a 2.5×–736× embedding-throughput advantage on the same H100. Its LLM retrieval tests used 82–415-document corpora in context, so those prices are lower-bound signals rather than a production quote. 1

Start with a shadow evaluation. Index with the public Octen-Embedding-8B checkpoint, then compare it with the smaller Qwen3-Embedding-0.6B checkpoint. Both model cards show direct embedding usage; Qwen's card also documents vLLM and Text Embeddings Inference paths. 23

Run the same query set through both routes. Measure hit@k, nDCG, task success, cost per query, p95 latency, memory, and failure cases. Keep an optional LLM reranker for the top 100 results, and keep the current path until your corpus shows that the extra reasoning pays for itself. 1