How do you choose an embedding model?

I start with our data, not a leaderboard. I collect one to two hundred real or realistic queries and label which chunks should come back for each. Public leaderboards such as MTEB help me shortlist three or four candidates, but their tasks rarely match our domain, language mix or document style.

For each candidate I index the same chunks and measure recall at 10 and mean reciprocal rank. Then I weigh the practical side. Does it support all our languages? Is the max input length long enough for our chunks? A 3072-dimension vector costs more storage and search time than a 768-dimension one, which matters at hundreds of millions of chunks. Hosted models are easy but send data out; open models need serving but give control. Some models expect different prefixes for queries and documents, and getting that wrong hurts results.

Finally I store the model name and version with the index. Embeddings from different models are not comparable, so a model change means re-embedding the whole corpus, and I plan that as a migration.