Best Hardware for Local LLMs: What to Buy
For most people building a local LLM setup, the best starting point is a desktop PC with an NVIDIA GPU offering 24 GB of VRAM, such as an RTX 4090 if you can find one at a sensible price. That much memory supports a useful range of quantized models, while NVIDIA’s broad software support makes setup easier; if you value low power, a smaller budget, or running very large models more than speed, other choices may fit better.
What to Look For
VRAM comes first for GPU inference. Model weights must fit in available memory to run efficiently on a GPU, and you also need room for the context window, runtime overhead, and the cache used to track the conversation. A 7B–8B parameter model can often run well in 8–12 GB with a suitable quantization; 12–16 GB is more comfortable, while 24 GB opens the door to many 30B–35B quantized models. Exact requirements depend on model, quantization, context length, and software. A model can sometimes be split between GPU and system RAM, but generation is usually slower.
Memory capacity and bandwidth both matter. System RAM is useful for loading models, running other applications, and CPU inference. Aim for 32 GB in a general-purpose desktop, or 64 GB if you plan to load large models partly into RAM. GPU memory bandwidth affects how quickly tokens are generated, especially for larger models; capacity still sets the practical ceiling.
Match the hardware to the workload. Chat and coding assistance benefit from responsive generation and enough context. Batch processing and serving several users need more memory and throughput. Fine-tuning is more demanding than inference and may require multiple GPUs, specialized techniques, or cloud hardware.
Check the whole system. Consider power draw, cooling, case clearance, motherboard slots, power connectors, operating-system support, and noise. For Apple systems, unified memory is shared by the processor and graphics, so it is not directly interchangeable with a discrete GPU’s dedicated VRAM.
NVIDIA GeForce RTX 4090 Desktop
Strengths: Its 24 GB of VRAM is a strong fit for local inference, and NVIDIA’s CUDA ecosystem is supported by many popular runtimes. It delivers high generation speed for models that fit in memory and is useful for other GPU workloads as well.
Trade-offs: The card is expensive, physically large, and power-hungry. Availability and pricing vary, and a new generation or used-market listing may change whether it is good value. Its 24 GB limit still rules out some larger models at higher precision or long context.
Best for: People who want a powerful single-GPU desktop and can justify the cost. Check current local prices and the exact card’s dimensions and power requirements before buying.
NVIDIA GeForce RTX 3090, Used
Strengths: Like the 4090, it has 24 GB of VRAM, making it a capable option for quantized models that need more memory than common 12–16 GB cards provide. A used card can offer substantial capacity for less than a newer flagship, depending on the market.
Trade-offs: It is older, draws considerable power, and is generally slower than a 4090. Used cards may have unknown histories, and warranty coverage can be limited. Inspect the seller’s return policy, card condition, and power supply compatibility.
Best for: Technically confident buyers who prioritize VRAM per dollar and are comfortable evaluating used hardware. It is not automatically a bargain: compare the final price, power costs, and condition with newer alternatives.
Apple Mac Studio with M-Series Ultra Chip
Strengths: Unified memory lets supported software allocate a large pool to model workloads, and the compact system combines CPU, GPU, memory, and storage in one quiet workstation. Configurations with 128 GB or more can accommodate some large quantized models that do not fit on a single consumer GPU.
Trade-offs: Unified memory is shared across the system, so not all of it is available to model weights. Performance and model support vary by software, and some CUDA-focused tools or workflows are unavailable or less straightforward. High-memory configurations are costly and generally cannot be upgraded later.
Best for: Users who want a tidy, low-maintenance desktop, already prefer macOS, or value capacity for larger models over maximum GPU speed per dollar. Confirm that your chosen inference software supports the specific chip and model format.
AMD Radeon RX 7900 XTX Desktop
Strengths: The 24 GB of VRAM provides useful capacity for local models, and the card can be appealing when its price is substantially below competing options. AMD’s ROCm software stack supports a growing selection of machine-learning tools.
Trade-offs: Compatibility is more dependent on operating system, driver, and runtime than with the most common NVIDIA setup. Some instructions and tools assume CUDA, so troubleshooting or adapting workflows may take extra effort. Verify that the exact software you plan to use supports your configuration.
Best for: Users comfortable checking software compatibility and tuning a Linux or Windows setup, especially when local pricing makes the card a better value.
Entry-Level NVIDIA GPU with 12–16 GB VRAM
Strengths: A lower-cost NVIDIA card with 12–16 GB can run many small and mid-sized quantized models, and benefits from broad CUDA compatibility. It is a practical way to learn local inference without buying a flagship GPU.
Trade-offs: The smaller memory pool restricts model size and context, and some workloads may run out of space once other applications or runtime overhead are included. Product names and memory amounts change between generations, so compare the actual VRAM—not just the model number or gaming performance—before purchasing.
Best for: Beginners, casual chat, and coding help with smaller models. If you expect to move to larger models soon, saving for a 24 GB card may be more economical than upgrading twice.
How to Choose
Start with your budget. If you already own a recent computer, try CPU inference or a modest GPU before replacing the whole system. For a new desktop, set aside money for adequate system RAM, storage, a reliable power supply, and cooling—not just the graphics card. Hardware prices fluctuate, so compare complete systems and current local listings rather than relying on a fixed price target.
Choose a target model size. For lightweight assistants and experimentation, 8–12 GB of VRAM may be enough. For more capable quantized models and room to experiment, 16–24 GB is a more flexible range. If your priority is a model too large for one GPU, consider a high-memory Mac, multiple GPUs, or CPU-and-RAM offloading, understanding that these options trade convenience or speed for capacity.
Think about context and speed. A long context can consume substantial additional memory, so a model that barely fits at a short context may not work for document-heavy tasks. Quantization reduces memory use, often with some quality trade-off. Smaller quantized models can feel faster and more responsive than a larger model forced to split across slow system memory.
Separate inference from training. Running a model locally is much easier than training one from scratch. Fine-tuning needs more memory and specialized tools; if that is your goal, check the requirements for the exact method and model before buying. For many personal projects, renting compute temporarily can cost less than owning hardware that sits idle.
Frequently Asked Questions
How much VRAM do I need for local LLMs?
About 8–12 GB can suit small quantized models; 16–24 GB gives more flexibility. Requirements rise with model size, precision, context length, and runtime overhead.
Can I run an LLM without a dedicated GPU?
Yes. CPU inference using system RAM works with compatible software, but it is usually slower. It can be a reasonable way to test small models or use hardware you already own.
Is 24 GB of VRAM enough for a 70B model?
Not for a typical 70B model entirely in GPU memory. Aggressive quantization and CPU or multi-GPU offloading may make some configurations possible, but performance and quality vary.
Is NVIDIA always the best choice?
No. NVIDIA is often the easiest choice for software compatibility, but an AMD card or Apple system may suit a specific budget, software stack, or memory-capacity need. Check support for your intended tools first.
How much system RAM should I get?
32 GB is a sensible general-purpose starting point; 64 GB is more useful if you expect CPU offloading, larger models, or several applications running together. More RAM does not replace dedicated GPU memory for fast GPU inference.
Bottom Line
For a straightforward, high-performance local LLM desktop, choose an NVIDIA GPU with 24 GB of VRAM—an RTX 4090 when its current price makes sense, or a carefully checked used RTX 3090 for a lower-cost route. If quiet operation and large shared memory matter more than CUDA compatibility or speed, consider a high-memory Mac Studio; if your needs are modest, start with a 12–16 GB NVIDIA card and a system with 32 GB of RAM.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.