Ollama Context Tuning: Move System Prompts & Long Workloads Safely
Ollama’s context window tuning lets you move system prompts and long workloads safely, but only if you understand the trade-offs between memory usage, speed, and model coherence. This article explains what context tuning does, how to adjust it for system prompts and long documents, and the practical limits you’ll hit with real hardware.
What Context Tuning Actually Does
Ollama runs large language models locally. The context window is the number of tokens (roughly 0.75 words each) the model can “see” at once. By default, many models use a 2048-token context, which is fine for short chats but too small for system prompts plus long documents. Tuning the context size (via the num_ctx parameter or the OLLAMA_CONTEXT_LENGTH environment variable) tells the model how much text to keep in memory while generating a response.
When you increase the context, the model can read more of your prompt, but it also uses more RAM and VRAM, and generation becomes slower. The key is finding the smallest context that fits your workload, not the largest your GPU can handle.
How to Set Context Size in Ollama
You can set the context size in three ways:
- Per-request: Add
num_ctxto the API request JSON (e.g.,"num_ctx": 8192). This overrides everything else for that call. - Model file: Create a Modelfile with
PARAMETER num_ctx 8192and runollama create mymodel -f Modelfile. - Global environment: Set
OLLAMA_CONTEXT_LENGTHbefore starting the server. This applies to all models unless overridden.
For system prompts, you don’t need a huge context—just enough to include the prompt plus the user’s message. For long workloads like summarizing a book, you need a context at least as large as the document plus the prompt and the output you expect.
Memory Requirements: The Real Constraint
Each token in the context uses memory for the model’s key-value cache. The exact amount depends on the model architecture, but a rough rule of thumb is about 1–2 MB per 1,000 tokens for a 7B model, and proportionally more for larger models. A 70B model with a 32k context can easily exceed 64 GB of RAM, which is why most users stick to 7B–13B models for long contexts.
If your system runs out of memory, Ollama will either crash or start swapping to disk, which makes generation painfully slow. You can check memory usage with nvidia-smi (GPU) or free -h (RAM) while a request is running.
Speed vs. Context: What to Expect
Longer contexts slow down generation for two reasons: the model has to process more tokens before it starts generating, and the attention mechanism becomes more expensive with each additional token. In practice, doubling the context can roughly double the time-to-first-token for a long prompt. Generation speed (tokens per second) also drops, especially on CPU-only setups.
For interactive use, keep context under 8k if possible. For batch processing, you can go higher, but be prepared to wait minutes for a single response.
System Prompts: Keep Them Short and Specific
A system prompt tells the model how to behave. It’s sent with every request, so if you set a huge context to accommodate a long system prompt, you waste memory on every call. Instead, keep system prompts under 500 tokens, and put the variable content (like a document) in the user message.
If you need a long system prompt (e.g., detailed instructions with examples), consider using a model that supports a separate system prompt slot, or truncate the prompt to the most essential parts. Ollama doesn’t have a separate system prompt cache, so the entire prompt is re-processed each time.
Long Workloads: Chunking vs. Full Context
When you have a document longer than your context, you have two options:
- Chunking: Split the document into pieces, summarize each, then combine summaries. This works with any context size and is more reliable for very long texts.
- Full context: Increase
num_ctxto fit the entire document. This preserves all details but requires more memory and may still hit the model’s maximum (often 32k or 128k for newer models).
Chunking is safer for most users because it avoids out-of-memory errors and lets you stay within the model’s optimal performance range. You can use a simple script to split text by paragraphs or fixed token counts, then pass each chunk as a separate user message.
Common Mistakes and How to Avoid Them
One common mistake is setting num_ctx higher than the model’s maximum. Ollama will accept the value but the model may produce garbled output or crash. Check the model card for the max context length (e.g., Llama 3 supports 8k natively, but can be extended with RoPE scaling).
Another mistake is forgetting that the output tokens also count toward the context. If you set num_ctx=4096 and your prompt is 3000 tokens, the model only has 1096 tokens left for the response. Plan for the output length.
Finally, don’t assume that a larger context always improves quality. Models trained on short contexts can lose coherence when forced to attend to very long inputs. If you see the model repeating itself or ignoring early parts of the prompt, reduce the context or chunk the input.
Frequently Asked Questions
Does increasing context size affect output quality?
Yes, but not always positively. Up to a point, more context helps the model remember details. Beyond that, the model may “forget” earlier tokens or become repetitive. The optimal context depends on the model and the task. For most models, 4k–8k is a sweet spot.
Can I use a context larger than the model’s training length?
Technically yes, if you use RoPE scaling (Ollama does this automatically for some models), but quality degrades. The model was trained with a specific positional encoding, and extrapolating beyond that can confuse it. It’s safer to stay within the official max.
What’s the difference between num_ctx and OLLAMA_CONTEXT_LENGTH?
num_ctx is a per-request parameter that overrides the default. OLLAMA_CONTEXT_LENGTH is a server-wide default that applies to all requests unless overridden. Use the environment variable for a consistent baseline and num_ctx for specific jobs.
How do I know how much memory my context will use?
Use the formula: memory ≈ (num_ctx × model_size_in_bits) / 8. For a 7B model with 16-bit weights, each token costs about 14 bytes for the KV cache. So 8k context uses ~112 MB. Add the model weights (14 GB for a 7B in 16-bit) and you get total RAM/VRAM. Monitor with system tools during a request.
Is it better to use a smaller model with a longer context or a larger model with a shorter context?
For most tasks, a larger model with a shorter context gives better quality. A 70B model with 4k context will understand nuance better than a 7B model with 32k. Use a long context only when the task truly requires it, like analyzing an entire book.
Can I move system prompts to a separate file to save context?
No, Ollama doesn’t support external system prompt files. You must include the system prompt in the request. However, you can use a Modelfile to embed a system prompt into a custom model, which is sent automatically and doesn’t count against your manual prompt tokens (but still uses context).
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.