How Much Memory Do You Need to Run LLMs Locally?

For local large language models (LLMs), plan on roughly 8 GB of available memory for small, quantized models, 16 GB for a useful range of 7–8-billion-parameter models, and 32 GB or more for larger models or comfortable multitasking. The exact requirement depends on model size, numerical precision, context length, and whether the model runs in GPU memory, system RAM, or both.

What “memory” means for a local LLM

People often use “memory” to mean several different things. A computer’s RAM is working memory shared by applications and, in some systems, the graphics processor. Dedicated VRAM is the memory on a discrete graphics card. An LLM needs space for its model weights and for temporary data used while generating text; the operating system and other applications need memory too.

Storage is different. A model file may occupy several gigabytes on an SSD, but loading it for inference requires working memory. A machine can have enough disk space to download a model and still lack enough RAM or VRAM to run it effectively.

There is no single minimum that applies to every model. A compact model in a compressed format may run on a modest computer, while a larger model, long conversation, or high-throughput setup can need many times more memory.

How model size translates into memory

Model names commonly include a parameter count, such as 7B or 70B: “B” means billion parameters. A simple first estimate is parameter count multiplied by the number of bytes used for each parameter. At 16-bit precision, that is about two bytes per parameter; at 8-bit, about one; at 4-bit, about half a byte.

• 7B at 16-bit: approximately 14 GB for weights alone.
• 7B at 8-bit: approximately 7 GB for weights alone.
• 7B at 4-bit: approximately 3.5 GB for weights alone.
• 13B at 4-bit: approximately 6.5 GB for weights alone.
• 70B at 4-bit: approximately 35 GB for weights alone.

These are rough decimal estimates, not guaranteed file sizes or complete runtime requirements. Quantization—the process of representing weights with fewer bits—also has metadata and implementation overhead. Formats and loading software differ, and some models use architectures that change the practical memory footprint. Leave headroom rather than treating the estimate as a precise minimum.

Practical memory ranges

The following ranges are starting points for one model running locally. They include some allowance beyond the weight file, but actual needs vary with software, model format, and workload.

• 8 GB system RAM: suitable mainly for smaller models, often with aggressive quantization and limited multitasking. A model may run slowly if it has to use system memory extensively.
• 16 GB system RAM: a practical entry point for many 7–8B quantized models, especially when the computer has no large GPU. Keep other memory-heavy applications closed.
• 32 GB system RAM: offers more room for larger quantized models, longer prompts, and ordinary desktop use alongside inference.
• 64 GB or more: can accommodate still larger models in system memory or provide room for demanding workloads, though more RAM does not automatically make generation fast.

For a GPU, compare its dedicated VRAM with the model’s actual runtime needs. A 7B model in 4-bit form may fit on a GPU with around 6–8 GB of VRAM, depending on context and software. A 13B 4-bit model commonly calls for roughly 10–12 GB or more to avoid tight memory limits. These are practical estimates, not universal compatibility guarantees.

Why context length adds memory

The prompt and conversation history are converted into tokens, and the model maintains a key-value (KV) cache for tokens it has processed. That cache grows as the context gets longer. Its size depends on the model’s architecture, number of layers, attention heads, token count, and cache precision; therefore, there is no reliable single “memory per token” figure for all LLMs.

A model that fits when answering a short prompt might run out of memory with a much longer document or chat history. Increasing the maximum context setting can reserve or consume more memory, though some software allocates the cache dynamically. If memory is tight, reduce the context length or use a model with a smaller memory footprint.

Choosing a setup that fits

Before downloading a model, check its parameter count, quantization, and the runtime’s stated memory guidance. Then compare the expected total with memory available to the application—not just the number printed on the computer’s RAM or graphics-card label.

1. Decide what you need: short chats and basic drafting are less demanding than processing long documents or running several requests at once.
2. Choose a model and quantization: smaller models and lower-bit weights use less memory, but aggressive quantization can affect output quality.
3. Check the hardware split: GPU-only inference needs sufficient VRAM; CPU inference uses system RAM. Some runtimes can split layers across GPU and RAM, but transfers between them can reduce speed.
4. Leave room for the rest of the computer: the operating system, runtime, KV cache, and other applications all compete for memory.
5. Test with your real workload: try the prompt length and context size you expect to use, then monitor RAM and VRAM while generating.

Memory versus speed and quality

Having enough memory means a model can load and run; it does not mean it will respond quickly. GPU compute capability, memory bandwidth, CPU performance, software, and how many layers fit in VRAM all affect generation speed. A model running entirely in system RAM can be useful, but it is often slower than one that fits on a capable GPU.

Quantization is a trade-off, not a free reduction. Lower-bit versions generally occupy less memory and may run on more hardware, but can reduce accuracy or change behavior relative to a higher-precision version. Model quality also depends on training and task fit, so parameter count alone does not determine which model is best.

Frequently Asked Questions

Can I run an LLM with 8 GB of RAM?
Yes, some small or heavily quantized models can run with 8 GB, but the operating system and other applications need memory too. Expect limitations in model choice, context length, and multitasking.

Is 16 GB RAM enough for a 7B model?
Often, if the model is quantized and the runtime’s other memory needs fit. A 16-bit 7B model’s weights alone are roughly 14 GB, so it generally will not fit comfortably within 16 GB of total system RAM.

Is VRAM more important than RAM?
For GPU inference, VRAM determines how much of the model and its working data can stay on the graphics card. System RAM still matters for the operating system and any layers or data kept off the GPU.

Does the model file size equal the RAM requirement?
No. The file size is a useful starting clue, but runtime overhead, the KV cache, temporary buffers, and software can push total use higher.

Does a longer context need more memory?
Usually. More tokens in the prompt and conversation require a larger KV cache, although the amount varies by model architecture and runtime.

Can I use system RAM when VRAM runs out?
Some tools can offload part of a model to system RAM, but this may make generation slower. Whether it works depends on the software, hardware, and available memory.

Related

• Indie Game vs. AAA Game: What’s Different in Practice?
• OpenClaw on a VPS vs Home Server: Which Should You Choose?
• NPU TOPS: What the Number Can and Can’t Tell You