Lexical Search vs Sparse Vector Search for Multilingual

For multilingual search, sparse vector search generally outperforms traditional lexical search because it handles vocabulary mismatch and cross-lingual semantics better. Lexical search (e.g., BM25) matches exact query terms against document terms, which fails when the same concept is expressed in different languages or with different word forms. Sparse vector search, by contrast, maps both queries and documents into a high-dimensional sparse space where related terms—across languages—receive non-zero weights, enabling retrieval without exact term overlap.

Formula

Lexical recall = (matched terms / query terms) × 100. This measures the fraction of query terms that appear verbatim in the document. Query term density = (matched terms / document length) × 100, showing how concentrated the matches are. Sparse vector search does not have a simple formula; it relies on learned term weighting and expansion.

Worked Example

Using the default values: query length = 5 words, document length = 100 words, matched terms = 2. Lexical recall = (2 / 5) × 100 = 40%. Query term density = (2 / 100) × 100 = 2%. This means only 40% of the query terms appear in the document, typical for cross-lingual queries where the rest are translated differently.

Background: Why Lexical Search Fails Multilingual

Lexical search, epitomized by BM25, treats documents as bags of words. It ranks documents by term frequency and inverse document frequency. For multilingual corpora, the fundamental issue is that the same concept is represented by different strings: “car� in English, “auto� in German, “voiture� in French. A query for “car� will not match a German document containing “auto� unless a synonym dictionary or translation is applied. Even with stemming, languages with rich morphology (e.g., Finnish, Turkish) produce many word forms that a simple stemmer may not unify.

Sparse vector search, often implemented with learned sparse representations like SPLADE or uniCOIL, overcomes this by expanding each token into a weighted set of related terms. For example, “car� might expand to “automobile�, “vehicle�, “auto�, and even German “auto�. The document vector also contains these expansions, so the query and document vectors share non-zero dimensions even without exact string match. This is still sparse—most dimensions are zero—but the non-zero weights are learned to capture semantic and cross-lingual relationships.

How Sparse Vector Search Works

In sparse vector search, each document is encoded into a vector where each dimension corresponds to a term in a large vocabulary (often subword tokens). The encoder, typically a transformer model, assigns a weight to each term based on its importance to the document. For queries, the same encoder produces a query vector. Retrieval then computes a similarity score, usually dot product, between the query vector and each document vector. The key is that the encoder has been trained on multilingual data, so it learns that “car� and “auto� are related across languages.

Compared to dense vector search (e.g., using embeddings from models like multilingual E5), sparse vectors are more interpretable: you can see which terms contributed to the match. They also require less memory for index storage because the vectors are sparse. However, they still require a neural model to generate the vectors, which is computationally expensive at index time.

Comparing Lexical, Sparse, and Dense for Multilingual

• Lexical (BM25): Fast, simple, no training needed, but fails on vocabulary mismatch and cross-lingual synonyms. Works well for same-language queries with exact terms.
• Sparse vector (learned): Handles synonyms and cross-lingual expansion, still interpretable, but requires model inference and more storage than BM25.
• Dense vector: Best for semantic similarity and cross-lingual transfer, but requires heavy computation and loses exact term matching (e.g., rare codes, product IDs).

In practice, many systems use a hybrid approach: combine BM25 with dense or sparse vectors to get both exact and semantic matches. For multilingual, sparse vectors often provide a good middle ground: better than BM25 on recall, cheaper than dense for large corpora, and more transparent.

How to Choose Between Lexical and Sparse

Consider your data and query patterns:

• If your corpus is monolingual and queries are short, well-formed, and use standard vocabulary, BM25 may suffice.
• If you have multilingual content or users query in multiple languages, sparse vector search is a strong upgrade.
• If you also need to handle paraphrases, long-tail synonyms, or queries that don’t share any terms with documents, consider dense vectors or a hybrid.
• If you have limited computational resources and need real-time indexing of millions of documents, BM25 is the cheapest. Sparse vector indexing requires GPU inference but is still cheaper than dense for storage.

Also evaluate your evaluation metric: if you care about recall@k (finding all relevant docs), sparse vectors generally beat BM25 on multilingual benchmarks like MIRACL. If you care about precision for exact identifiers (e.g., part numbers), lexical may be better.

Common Problems and Misconceptions

A common misconception is that sparse vector search is just “BM25 with synonyms�. In reality, the expansions are learned from data, not manually curated, and they capture context-dependent relationships. Another issue is that sparse vectors can be large if the vocabulary is huge; pruning is needed to keep index size manageable. Also, training a sparse encoder requires paired query-document data, which may not be available for low-resource languages.

Another problem is that lexical search can be improved with simple techniques like language detection and translation, but this adds latency and error. Sparse vectors avoid that but require a multilingual model, which may not cover all languages equally. For very low-resource languages, both approaches may underperform.

Costs and Practical Considerations

Sparse vector search has higher indexing cost than BM25 because you must run a neural model on every document. For a corpus of 10 million documents, this could take hours on a GPU. Storage is also higher: each document vector may have hundreds of non-zero terms, each requiring an integer ID and float weight. In contrast, BM25 stores only term frequencies and document lengths. Query latency is similar, but the neural encoder adds a few milliseconds per query.

For multilingual, you also need to consider language coverage. Models like XLM-R are trained on many languages, but performance varies. If your corpus includes languages not well represented, you may need to fine-tune on your own data.

Safety and Accuracy Notes

When implementing sparse vector search, be careful with tokenization: subword tokenizers can split words across languages inconsistently. Also, ensure your evaluation uses appropriate multilingual metrics, such as nDCG@10 averaged per language. Avoid assuming that a model trained on one language pair will generalize to all. Always test on your own corpus with a held-out set.

Frequently Asked Questions

Q: Is sparse vector search always better than lexical for multilingual?
A: No. For exact matches, like product codes or proper names, lexical can be better. Sparse vectors excel at semantic and cross-lingual matching, but if your queries are always in the same language as the documents and use standard terms, BM25 might be sufficient and much cheaper.

Q: Can I use sparse vectors without a GPU?
A: Yes, for inference you can use CPU, but it will be slower. Indexing a large corpus on CPU may be impractical. You can also use pre-computed sparse vectors from a service, but that adds cost.

Q: How do I combine lexical and sparse search?
A: Use a hybrid retrieval approach: run both BM25 and sparse vector search, then merge results using Reciprocal Rank Fusion (RRF) or a learned weight. This often improves recall while preserving exact-match precision.

Q: What is the difference between sparse and dense vectors?
A: Sparse vectors have mostly zero values and are interpretable; dense vectors are low-dimensional and every value is non-zero. Dense vectors capture deep semantics but lose term-level evidence. Sparse vectors are a compromise.

Q: Do I need to translate queries for lexical search?
A: Not if you use sparse vectors. For lexical, you would need to translate or expand queries with synonyms, which is error-prone and doesn’t cover all languages. Sparse vectors learn cross-lingual alignments automatically.

Related

• Star Trek Commander Reprint Value: Best Deck to Buy
• Plex Re-Encoding: Can It Free Half a Drive?
• Delete Duplicate Excel Files From PC: Full Guide

in q | Query length (words) | words | 5 | 1
in d | Document length (words) | words | 100 | 1
in m | Matched terms | terms | 2 | 0
out recall | Lexical recall (matched/total) | % | (m / q) * 100 | 1
out density | Query term density | % | (m / d) * 100 | 2