Why a 100K-Token Prompt Can OOM After Model Load

Updated 5 November 2026 · Standard formula, shown below

A 100,000-token prompt can cause an out-of-memory (OOM) error even after a model loads because loading the model’s weights is only one part of its memory use. Processing a long prompt also needs memory for the key-value (KV) cache, temporary computations, and runtime overhead; if those exceed available GPU or system memory, the process can fail during prompt ingestion.

Model weights are only the starting cost

Model loading places its parameters—the learned weights—into memory. Their size depends on the number of parameters and their numerical precision or quantization. For example, lower-precision weights generally take less space than full-precision weights, though exact savings depend on the format and implementation.

That loaded footprint is not the total footprint. A running inference process may also need memory for the KV cache, intermediate activations, attention calculations, execution workspaces, and framework or driver allocations. Some of these allocations vary with the prompt, batch size, output length, software, and hardware. A model that just fits when idle can therefore run out of memory as soon as it starts handling a large request.

Why a long prompt needs a KV cache

During generation, a transformer processes tokens in sequence. To avoid recomputing all earlier attention information for every new token, many inference systems store keys and values from previous tokens in a KV cache. The cache grows as tokens are processed and must remain available while the model continues generating.

Its memory use is affected by the number of tokens, the number of layers, the attention configuration, and the cache’s data type. A simplified estimate for a conventional transformer cache is:

KV cache bytes ≈ 2 × layers × tokens × KV heads × head dimension × bytes per value

The factor of two accounts for keys and values. This is a rough estimate, not a universal accounting formula: grouped-query attention, sliding-window attention, cache quantization, tensor parallelism, and implementation details can change what is stored and where. Still, it shows why context length matters. If the token count grows tenfold and the other factors stay the same, the cache can grow roughly tenfold as well.

A 100K-token prompt is also not the same as a 100K-character prompt. Tokens are units used by a tokenizer; their relationship to characters varies by language and content. Code, unusual symbols, and some non-English text may tokenize differently from ordinary prose.

Prompt ingestion can create a temporary peak

Before generating an answer, the model must process the prompt, a stage often called prefill or prompt ingestion. The system may perform large computations across many input tokens and allocate temporary buffers for attention and other operations. Those buffers can push peak memory above the amount needed for the final KV cache alone.

The exact peak depends on the attention implementation and serving software. Some approaches avoid materializing a full token-by-token attention matrix, reducing memory use; others may require more temporary space. Processing a prompt in chunks can also change the peak, although it does not eliminate the memory needed to retain the context that the model will use later.

This explains a common misconception: a model successfully loading does not prove that it can accept every prompt within its advertised context window. A context limit describes the sequence length the model or service can support under stated conditions; it does not guarantee that every hardware configuration has enough memory for that length, batch size, cache format, and output allowance.

Other factors that push memory over the limit

Output length: Generated tokens extend the cache. A prompt that barely fits may fail later if there is no room for the requested completion.
Batching and concurrent requests: Multiple sequences can require multiple caches. Servers may allocate memory for simultaneous users or reserve capacity for scheduling.
GPU memory already in use: Other processes, graphics workloads, or another model can reduce the space available to inference.
Memory fragmentation and reservations: Available memory in total is not always available as one suitably sized allocation. Frameworks and drivers may also reserve memory.
Model and cache formats: Quantizing weights does not necessarily quantize the KV cache. A model can have compact weights but still use a substantial cache.
Offloading limits: Moving some weights or cache data to system RAM can relieve GPU pressure, but it needs enough host memory and may slow processing. It cannot help if the combined system resources are insufficient.

How to diagnose and reduce OOM failures

First identify when the failure occurs. An error while loading points toward weight placement or insufficient memory for the model itself. A failure as the prompt is submitted suggests prompt-processing or cache pressure. A failure partway through generation often means the cache grew until it crossed the available limit, though logs are needed to distinguish that from other causes.

Check actual free memory on the GPU and host while the process runs, not just the advertised total capacity. Look for other workloads and memory reserved by the inference runtime.
Reduce the prompt length by removing repeated instructions, irrelevant history, or duplicated documents. Retrieve only the passages needed for the current question rather than placing an entire collection in the prompt.
Leave room for the answer. The usable sequence budget must accommodate both input and generated tokens. Follow the serving system’s definition of its context limit, since accounting and reserved space can differ.
Lower concurrency or batch size if the error occurs under load. This can reduce simultaneous cache and workspace allocations, though it may reduce throughput.
Use a compatible memory-saving configuration, such as a smaller model, a supported cache format, or chunked prefill where available. Verify the trade-offs and actual settings in the software documentation.

Shortening a prompt is often the simplest test: if a substantially shorter input succeeds under the same conditions, context-related memory pressure is likely. It is not definitive proof, since changing prompt length can also alter temporary allocation patterns.

Context limits, memory limits, and practical trade-offs

Context capacity and memory capacity are related but different. A model may support a long context in principle, while a particular device can serve only shorter prompts at a chosen precision and concurrency. Conversely, fitting the weights does not mean the device has enough headroom for a large cache.

There is no single memory figure for “100K tokens.” The answer depends on architecture, layer and attention dimensions, cache data type, output budget, batch size, runtime, and hardware. An estimate based only on model parameter count misses the token-dependent cache and transient work. Use estimates as planning tools, then measure peak usage with the intended model and serving configuration.

Frequently Asked Questions

Does a 100K context window mean 100K tokens will always fit?
No. It indicates a supported sequence length under some configuration, not a guarantee for every device, batch size, cache type, or output length.

Why does the error happen after the model loads?
Loading mainly accounts for the weights. Prompt processing adds cache and temporary memory allocations, which can exceed the remaining capacity.

Will quantizing the model fix the problem?
It may reduce weight memory, but the KV cache and temporary buffers can still be large. Whether it helps enough depends on the model and inference setup.

Can I fix an OOM by reducing the output limit?
It can help when generation would otherwise grow the cache beyond available memory. It may not solve an OOM that happens during prompt ingestion.

Does splitting a prompt into chunks eliminate the memory problem?
Chunked processing can reduce some temporary peaks, but if the system retains all processed context in its cache, the cache still grows with the total retained tokens.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.