Semantic Search Without an Answer-Generating LLM

Updated 10 November 2026 · Standard formula, shown below

Semantic search finds documents by meaning rather than relying only on exact keyword matches. You can build it without an answer-generating large language model (LLM): a text embedding model converts queries and documents into vectors, and a vector index retrieves the closest matches. You will need a collection of searchable text and a way to run an embedding model and store or search vectors.

What You’ll Need

A set of documents or records with text users can search, plus stable identifiers and any useful metadata, such as date, category, or access permissions.
An embedding model that accepts text and returns vectors. It can run locally or through an embedding service; neither option needs to generate answers.
A vector search library or database that supports nearest-neighbor search. For a small prototype, a local vector index may be enough; larger or frequently updated collections may need a database.
A development environment and a way to run the indexing and search code. Check model and library documentation for installation, input limits, and vector dimensions.
A small set of representative test queries and expected relevant documents, so you can judge whether retrieval works.

Step-by-Step

Define what counts as a good result. Decide what users will search, what a result should contain, and how many results to return. For example, a result might show a document title, a short excerpt, and a link to the source. Write down several realistic queries and the records a knowledgeable reviewer would expect to see. This gives you a baseline beyond whether the system merely returns something.
Prepare and identify your source text. Extract readable text from your documents and preserve a stable ID for each source. Remove repeated navigation, boilerplate, and irrelevant markup, but keep details that affect meaning. Store metadata separately or alongside each text segment. If the content has access restrictions, plan how to apply them during retrieval; a vector match must never override the user’s permissions.
Choose one embedding model and keep it consistent. Select a model suited to your language and content, considering its input length, hosting requirements, and output size. An embedding is a numerical representation of text, not an answer. Use the same model and compatible preprocessing for documents and queries. If you later change models, regenerate the stored document vectors before comparing them with new query vectors.
Split long documents into useful chunks. Embedding a whole book or a very long page can blur specific topics or exceed the model’s input limit. Split text into paragraphs or chunks that preserve enough context to make each passage meaningful. A common starting point is a few hundred words per chunk, with a small overlap where sentences or sections might otherwise be cut apart. Treat that as a testable starting point, not a universal rule. Keep the source ID, chunk number, and relevant metadata for every chunk.
Create and store document vectors. Send each chunk to the embedding model and save the returned vector with its text, source ID, and metadata. Check that every vector has the expected dimension and that records are matched to the correct text. For a small collection, you can calculate vectors in a batch and save them in a local index. For a larger or changing collection, use an index that supports updates, deletion, and the expected query load.
Implement the search path. When a user submits a query, apply the same basic text cleanup and create its embedding with the same model. Search the vector index for the nearest stored vectors, using the similarity measure supported by the model and index. Return the top few matching chunks with their source details. This is retrieval only: show the passages or links directly rather than asking an answer-generating model to write a response.
Apply filters and present results clearly. Use metadata filters for constraints such as category, date, or permissions, preferably as part of the index query so excluded records are not surfaced. Display enough context for users to assess each match, such as a title and relevant excerpt. Link to the original source when possible. Avoid presenting a similarity score as a probability that the result is correct; scores are useful for ordering, but their scale depends on the model and index.
Evaluate with real queries and tune one thing at a time. Compare the top results against your expected matches. Check whether relevant documents appear near the top, whether results are duplicated, and whether unrelated passages outrank useful ones. Adjust chunk size, overlap, number of results, or filters one at a time, then rerun the same tests. Add queries that expose misses, including abbreviations, alternate wording, and queries with several distinct concepts.
Plan for updates and failures. When a source changes, re-embed its affected chunks and replace the old vectors; remove vectors when a source is deleted. Record model and preprocessing versions so you know which vectors must be rebuilt after changes. Handle empty queries, model or index timeouts, and no-result searches with clear messages. Monitor response time and indexing errors as the collection grows.

Common Mistakes

Mixing vectors from different embedding models is a frequent source of poor results: their dimensions or vector spaces may differ, so comparisons are not meaningful. Another mistake is embedding oversized documents as one unit, which can hide the passage that answers a specific query. At the other extreme, tiny chunks can lose context and produce scattered matches.

Do not assume semantic search replaces keyword search in every case. Exact names, product codes, and uncommon identifiers may be handled better by lexical matching. Also avoid treating the nearest result as necessarily relevant, using similarity scores as calibrated confidence, or returning matches before enforcing permissions and metadata constraints.

Tips

Start with a small, representative collection and a simple exact vector search before optimizing for scale. Keep the original text and metadata available so results can be inspected and re-indexed. Use a fixed evaluation set when changing models or chunking, and track whether expected documents appear in the first few results.

For mixed search needs, combine vector retrieval with keyword search and merge or rerank the candidate lists. This can help with both conceptual queries and exact terms, without generating an answer. Make the interface explain that it shows matching sources, not a synthesized response.

Troubleshooting

Results are unrelated: Confirm that query and document vectors come from the same model, use the same preprocessing, and have compatible dimensions. Check whether the query is too vague and whether chunks contain enough context. Inspect the actual top matches rather than relying only on scores.

Good documents are missing: Verify that the source text was extracted and indexed, that the query is not being over-filtered, and that the result limit is not too small. Test alternate wording and review chunk boundaries. Add keyword retrieval if the missing query depends on an exact code or name.

Search is slow: Measure query embedding time separately from vector lookup. Reduce unnecessary text processing, check index configuration, and consider a search system designed for your collection size and traffic. Do not switch to approximate search without checking whether its speed-versus-recall tradeoff is acceptable.

Updates appear stale: Confirm that changed documents were re-embedded and old chunks removed or replaced. Check for duplicate IDs and failed indexing jobs. Keep an update log so you can trace a result to its source version.

Frequently Asked Questions

Can semantic search work without an LLM?
Yes. It needs an embedding model to represent text as vectors, but it does not need a model that generates answers. The search system returns matching source passages.

Is an embedding model still a language model?
It may be a model trained on language, but its role here is to produce vectors, not write a response. You can run it locally or use an embedding service.

How large should document chunks be?
There is no single best size. Start with a few hundred words, preserve paragraph context, and test against representative queries. Respect the embedding model’s input limit.

Do I need a vector database?
Not necessarily. A local vector index can suit a prototype or modest, mostly static collection. A dedicated database can help with larger collections, frequent updates, filtering, or multiple users.

Can I search by exact keywords too?
Yes. Combine vector results with keyword or full-text results when exact names, codes, or phrases matter. Evaluate the combined ranking using the same test queries.

How do I know whether the results are good?
Use a set of realistic queries with known relevant sources. Check whether those sources appear near the top, inspect misses, and repeat the evaluation after each significant change.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.