Better Vector Search for Long Documents: Chunking in Manticore
This article explains how to improve vector search for long documents by chunking inside Manticore Search. You’ll learn what chunking is, why it matters for retrieval quality, and how to configure Manticore’s built-in chunking options to get better results.
Background: Why Chunking Matters for Vector Search
Vector search works by converting text into embeddings—dense numerical vectors that capture semantic meaning. When you search, the system compares the query’s vector to document vectors using cosine similarity or another distance metric. The problem with long documents is that a single embedding for a 10,000-word article averages out the meaning of individual sections. A query about a specific detail buried in the middle will return a low similarity score because the overall vector is diluted by all the other topics.
Chunking solves this by splitting a long document into smaller pieces, each with its own vector. When you search, the system finds the most relevant chunk, and you can then map that chunk back to its parent document. This is the standard approach used by retrieval-augmented generation (RAG) systems and modern search engines. Manticore Search, an open-source search engine, added vector search support in version 6.3.0 (2023) and introduced built-in chunking in version 6.3.2 (2024).
How Manticore’s Chunking Works
Manticore’s chunking feature is integrated into the indexer. When you define a table with a vector column, you can specify a chunking option in the source or table definition. The indexer then splits the text field into chunks, embeds each chunk, and stores the vectors along with metadata that links chunks to their parent document.
Here’s a minimal example of a table definition:
CREATE TABLE articles (
id INTEGER,
title TEXT,
body TEXT,
body_embedding VECTOR(384) CHUNKED BY body
) chunking = (
type = 'source',
source = 'body',
target = 'body_embedding',
chunk_size = 512,
overlap = 64
);In this example, the body field is chunked into pieces of 512 tokens (using the tokenizer’s token count) with an overlap of 64 tokens between consecutive chunks. The CHUNKED BY clause in the column definition tells Manticore that the vector column is derived from the body field. The indexer will generate one vector per chunk, not one per document.
When you query, you use the same KNN operator as usual. By default, Manticore returns the most similar chunks. To get whole documents back, you can group by the parent document ID or use the GROUP BY clause. For example:
SELECT id, title, body FROM articles
WHERE KNN(body_embedding, QUERY('What is the capital of France?'))
GROUP BY id
ORDER BY _score DESC
LIMIT 5;This returns the top 5 documents, each represented by its best-matching chunk.
Choosing the Right Chunk Size and Overlap
The two main parameters are chunk_size and overlap. There’s no one-size-fits-all, but here are practical guidelines:
- Chunk size (tokens): For general text, 256–512 tokens works well. Smaller chunks (128–256) capture finer details but increase the number of vectors and may hurt performance. Larger chunks (512–1024) preserve more context but can dilute meaning. Consider your embedding model’s maximum input length—many models accept up to 512 tokens, so chunk_size should not exceed that unless your model supports longer sequences.
- Overlap (tokens): Overlap helps maintain continuity between chunks, especially when a sentence or concept spans the boundary. A common choice is 10–20% of the chunk size. For 512-token chunks, 64–100 tokens of overlap is reasonable.
- Document type: For structured documents with clear sections (e.g., legal contracts, research papers), you might chunk by section headers instead of fixed token counts. Manticore doesn’t directly support semantic chunking (e.g., by paragraph), but you can pre-process your text before indexing to insert special delimiters that the tokenizer can use as chunk boundaries.
Common Problems and Misconceptions
Misconception: “Chunking is automatic after I enable it.� Actually, you must specify the source field and the target vector column. If you don’t, Manticore treats the whole document as one vector.
Problem: “My search results got worse after chunking.� This often happens because the default chunk size is too small or too large for the content. If chunks are too small, a query may match a chunk that is only tangentially relevant. If too large, you’re back to the original problem. Tune chunk_size and overlap based on your data. Also, check that your embedding model is appropriate for the chunk length—some models are trained on sentence-level text and may produce poor embeddings for longer chunks.
Problem: “How do I get the whole document back?� As shown above, use GROUP BY on the parent document ID. Manticore stores a _chunk_id and _parent_id internally; you can access them in your query to filter or group.
Comparison: Manticore vs. Other Vector Databases
Manticore’s built-in chunking is relatively unique—many vector databases (e.g., Pinecone, Weaviate, Qdrant) require you to chunk documents yourself before ingestion. Manticore’s approach simplifies the pipeline: you send raw text, and the indexer handles splitting and embedding. This is convenient but gives you less control over chunk boundaries. If you need custom chunking logic (e.g., by paragraphs or semantic sections), you’ll still need to pre-process text outside Manticore.
Performance-wise, Manticore uses an HNSW (Hierarchical Navigable Small World) index for approximate nearest neighbor search, which is fast and memory-efficient. However, the built-in chunking increases the number of indexed vectors, so you’ll need more RAM. For very large corpora, you might consider a dedicated vector database, but for most applications, Manticore’s all-in-one approach (full-text + vector + SQL) is a strong advantage.
Costs and Resource Considerations
Chunking increases the number of vectors stored, which directly impacts memory usage. Each vector is typically 4 bytes per dimension (float32). For a 384-dimension vector, that’s about 1.5 KB per vector. If you have 100,000 documents with an average of 10 chunks each, that’s 1 million vectors, using roughly 1.5 GB of RAM just for the vectors (plus overhead for HNSW graphs).
CPU usage also increases during indexing because the embedding model must process each chunk. If you’re using a local embedding model, consider batching and GPU acceleration. Manticore itself doesn’t embed—you need an external embedding service or a plugin. The chunking happens before embedding, so the text is split first, then each chunk is sent to the embedding model.
Alternatives to Built-in Chunking
If Manticore’s chunking doesn’t fit your needs, you have alternatives:
- Pre-chunking outside Manticore: Write a script that splits your documents into chunks, embeds them, and indexes each chunk as a separate document with a common parent ID. This gives you full control over chunk boundaries and allows you to use any embedding model.
- Use a different vector database: If you need advanced chunking strategies (e.g., recursive character splitting, semantic splitting), tools like LangChain can pre-process text for any vector store.
- Hybrid search: Combine Manticore’s full-text search with vector search. For long documents, you can index the whole document for keyword search and also store chunk-level vectors for semantic search, then fuse results.
Frequently Asked Questions
Q: What is the best chunk size for Manticore vector search?
A: It depends on your embedding model and content. For most general text, 256–512 tokens is a good starting point. If your model has a 512-token limit, use 512 or less. Test with a small sample and evaluate retrieval quality on your own queries.
Q: How do I get the original document back after searching chunks?
A: Use GROUP BY on the parent document ID. For example, SELECT id, title, body FROM articles WHERE KNN(body_embedding, QUERY('...')) GROUP BY id. This returns the best-matching chunk per document, but you can also fetch the full document by joining back to the original table.
Q: Does Manticore support semantic chunking (e.g., by paragraph)?
A: Not directly. Manticore chunks by token count with optional overlap. For semantic chunking, you need to pre-process your text and insert custom delimiters that the tokenizer can use as chunk boundaries, or chunk externally and index each chunk as a separate row.
Q: Will chunking slow down my search?
A: Search speed depends on the number of vectors. More chunks mean more vectors, which can increase memory usage and slightly slow down KNN search. However, Manticore’s HNSW index is efficient, and for most datasets the impact is minimal. Proper tuning of chunk_size and overlap can balance quality and performance.
Q: Can I use chunking with any embedding model?
A: Yes, Manticore doesn’t care which model you use as long as you provide the embedding vectors. You can use any model that accepts text and returns a fixed-size vector. The chunking happens before embedding, so each chunk is embedded separately.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.