How to Evaluate Embedding Models on Your Own Data Before Choosing One for RAG

When you build a retrieval-augmented generation (RAG) system, it searches your documents and gives useful passages to a language model before the model writes an answer. An embedding model turns text into numbers that capture its meaning. A retrieval service compares those numbers to find passages related to a user’s question.

The best model for your system depends on your documents, questions, passage boundaries, and limits on response time and cost. Public benchmark scores help you choose models to test, but they do not show which one will retrieve the right evidence from your data. Compare candidates on a labeled set of realistic questions, first under the same retrieval conditions and then in the RAG configuration you plan to use.

Build a test set from real questions and evidence

Start with questions people actually ask, such as privacy-safe support tickets or search logs. Include short questions and questions with ambiguous wording. Also test queries with acronyms or specialized terms, multi-part questions, and queries in relevant languages. Also include questions your documents cannot answer. Without those, your test cannot show whether retrieval returns plausible-looking passages when the needed evidence is absent.

For each question, mark the passages or specific sections of text that support an answer. A question may have several valid sources, so do not mark only one passage as correct when several passages provide relevant evidence. Ask a subject-matter reviewer to check ambiguous questions and specialized material.

If you have few real questions, create candidate questions from a sample of your documents. Check that some sound like real questions and that the cited passages support them. A Chroma technical report describes using generated queries to compare embedding models in a documentation-bot case study. A separate study of synthetic RAG evaluations found that synthetic tests ranked retriever configurations in line with human-labeled evaluations across four datasets, but did not consistently rank systems with different generator architectures. Use synthetic questions to expand the test set, while still checking real queries.

There is no established universal number of questions that makes an evaluation reliable. If the models’ results vary widely from question to question, collect more examples, especially in categories where the difference matters.

Compare models under controlled conditions

For the first comparison, change the embedding model while keeping the rest of retrieval fixed. Use the same snapshot of your document collection, passage boundaries, and questions. Keep filters, result count, and vector search settings fixed. Disable reranking and hybrid search if you want to isolate the embedding model’s effect. Hybrid search combines semantic matching with keyword matching; a reranker is a separate component that reorders retrieved passages.

Create embeddings from the same passages again for each model. Follow the model provider’s instructions for encoding queries and documents, and use its recommended similarity measure. For example, Cohere’s embedding documentation specifies different ways to encode search queries and documents in its embeddings setup. That example applies to the documented Cohere models, so check the instructions for each model you test.

Record the exact model name and version, input settings, vector dimensions, preprocessing, and evaluation date. These details let you repeat the comparison and help explain changes after a provider updates a model.

A controlled comparison should not erase model-specific instructions. Instead, use each candidate as its provider documents it, while keeping the corpus, queries, and retrieval conditions constant. Then run a second, separate comparison in which you tune the shortlisted candidates for the intended RAG system. If you change chunking or add a reranker in that second test, report those changes so system-level improvements do not look like embedding-only gains.

Choose metrics that match retrieval failures

No single score captures every retrieval problem. Start with these measures:

• Recall@k: The share of labeled relevant passages found among the top k results. This shows whether retrieval finds enough evidence.
• Precision@k: The share of the top k results that are relevant. This shows how much distracting material enters the context.
• MRR, or Mean Reciprocal Rank: Measures how high the first relevant result ranks across questions.
• nDCG@k: A ranking measure that accounts for different degrees of relevance and rewards highly relevant results near the top.

Choose k to reflect how many passages your RAG system sends to the language model. If the answer needs several pieces of evidence, emphasize recall. If the system supplies only a few passages, ranking and precision matter because irrelevant results consume space that could hold useful evidence. Microsoft’s retrieval evaluation guidance describes these metrics and recommends looking at more than one measure.

When passage boundaries and the amount of text the language model can read matter, measure how much relevant text appears in retrieved passages, since passage-level scores can miss the exact evidence an answer needs. Chroma’s work on chunk evaluation discusses token-level coverage and overlap measures for this purpose. Keep ranking metrics as well when your system has a limited context budget or when the order of passages affects the answer.

Compare results by query category as well as overall averages. A model may handle questions with exact product names well, while another may retrieve paraphrased questions better. Overall scores can hide these differences.

Include lexical retrieval as a baseline

Test at least one keyword-based search method alongside the embedding models, such as BM25, a method for ranking keyword matches. Exact identifiers and rare terms can favor lexical matching. The same is true of codes and dates. A win for keyword search in a category points to a retrieval issue worth investigating; by itself, it does not establish that one embedding model is best.

The BEIR benchmark found BM25 to be a robust baseline across its evaluated domains. It also reported that some high-performing reranking methods and methods that compare query and document parts during later scoring required more computation. Those results support comparing both retrieval quality and resource demands; they do not predict which method will lead on your corpus.

If your production design may combine keyword and vector search, test that combination after the isolated comparison. Keep its results separate from the embedding-only results, since the combination changes more than the embedding model.

Test answer quality and operating costs separately

Good retrieval scores do not guarantee good generated answers. A language model can misuse evidence even when retrieval finds it, and a retrieved passage can be truncated before the model sees it. For shortlisted configurations, test the full RAG system using the same generation model, prompt, and context budget where feasible. Assess answer relevance, completeness, and whether the answer stays grounded in retrieved evidence. Microsoft’s RAG evaluation guidance separates retrieval evaluation from evaluation of the generated response.

When an answer fails, inspect where the failure occurred. The needed evidence might be missing from search results, too far down the list, cut off when the system assembles the context for the generator, or present but misused by the generator. This diagnosis helps you avoid blaming the embedding model for a problem elsewhere in the pipeline.

Also measure how long it takes to create document and query embeddings and build the index. Record the number of values in each vector, how much storage the index uses, and any usage costs. Compare how many requests each model handles over time, and track failures such as rate limits if they affect your expected traffic. If two candidates have similar retrieval quality, these measurements help determine which one meets your operating limits.

Choose against requirements, not one headline score

Use public benchmarks to identify candidates worth testing, then make the decision from your own results. The original MTEB study covered multiple task types and found that no embedding method led across them all. Its findings make benchmark scores useful context, not a guarantee for a particular corpus.

Set minimum requirements before you compare results. For example, decide how much evidence retrieval must find at the context depth your system supports, and what latency or cost the service can tolerate. Then compare candidates across important query categories and check how stable their results are. When results vary widely or rankings remain uncertain, collect more evaluation data before choosing a winner.

A useful report records the model and settings, document collection snapshot, passage boundaries, test questions, number of retrieved passages, metrics, and operating measurements. It should also state how the production-like test differed from the controlled comparison and identify thinly tested areas, such as rare languages or questions without real user examples.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com