Speed Up Local LLM Inference on a CPU-Only PC

Updated 5 November 2026 · Standard formula, shown below

You can make a local language model respond faster on a CPU-only PC by choosing a smaller, efficiently quantized model, using CPU-optimized software, and avoiding unnecessary work per request. The biggest gains usually come from fitting the model comfortably in memory and matching its settings to your processor; no single setting makes every model faster.

What You’ll Need

A local LLM runtime that supports CPU inference, such as llama.cpp or a desktop application built on it.
A model file in a format supported by your runtime. Quantized GGUF models are commonly used with llama.cpp-based tools.
Your PC’s operating system, processor model, available RAM, and free disk space. Check these before downloading a model.
A short, repeatable prompt to use when comparing settings.

Step-by-Step

Measure your starting point. Run your current model with a representative prompt and note how long it takes to produce an answer. If the application reports tokens per second, record the prompt-processing rate and generation rate separately: they measure different stages. Keep the prompt, output length, and settings the same for later tests.
Check whether memory is the bottleneck. Look at RAM use while the model is loaded and generating. If available memory is nearly exhausted, the operating system may move data to disk, causing severe slowdowns. Close memory-heavy applications and use a smaller model or lower-memory quantization. Leave room for the operating system, the runtime, and the prompt’s context; do not choose a model solely because its file size appears to fit.
Choose a smaller model before tuning obscure settings. For interactive use, try a model with fewer parameters or a smaller, distilled model in the same task category. A smaller model generally generates faster, though its answers may be less capable. Compare answer quality on your own typical questions, not just a sample benchmark.
Use a suitable quantized model. Quantization stores model weights with fewer bits, reducing memory use and often improving CPU speed. In supported GGUF models, common labels include Q4 and Q5; lower-bit choices are usually smaller, while higher-bit choices may preserve more quality. Naming conventions and quality vary by model, so compare supported versions from a trustworthy source. Avoid converting or mixing files unless you know the runtime supports that model architecture and format.
Install a CPU-optimized runtime build. Use the runtime’s current instructions for your operating system and processor. Some builds can use CPU instruction sets such as AVX2 or AVX-512 when the processor supports them. A build compiled for unsupported instructions may fail to start, so use a compatible build rather than forcing an instruction option. Keep the runtime updated through its normal release or package channel.
Set thread count sensibly. Start with the runtime’s automatic CPU setting, or try a number close to your physical core count. Test a few nearby values rather than assuming that every logical thread helps: too many threads can compete for memory bandwidth or slow other work. Change one setting at a time, then compare generation speed and responsiveness under the same conditions.
Reduce context length to what you need. The context window covers the prompt and the conversation history, not just the answer. A very large context can increase memory use and slow prompt processing. Set a smaller limit if your tasks are short, and remove old conversation turns when they are no longer relevant. Do not set it below the size required for your actual prompt.
Limit unnecessary output and prompt work. Ask for a concise response when that suits the task, and set a reasonable maximum output-token limit in the runtime. Long system instructions, pasted documents, and large chat histories take time to process. Keep only useful context. A shorter prompt often helps the time before the first generated token, while a shorter requested answer reduces total generation time.
Use power and system settings that allow sustained CPU work. On a laptop, connect the charger and select an appropriate performance or best-performance power mode if available. On a desktop, close background workloads that compete for the CPU. Keep ventilation clear; thermal throttling can reduce speed after the processor heats up. Do not disable thermal protections.
Benchmark one change at a time. Restart the runtime when a model or launch option requires it, then run the same prompt several times. Ignore an unusually slow first run if the runtime is loading files or building caches, but do not compare a warmed-up run against a cold start without noting the difference. Keep settings that improve real-world response time without unacceptable quality loss.

Common Mistakes

Picking the largest model that barely fits. A model that pushes the PC into swap can be slower than a smaller one that stays in RAM.
Assuming more threads always means more speed. Threading helps only when the runtime, processor, and workload benefit from it.
Changing several settings at once. You will not know which change helped or caused a regression.
Comparing different prompts or output lengths. Token counts and prompt complexity affect timing, so keep the test consistent.
Confusing model loading with generation. A slow first launch may reflect disk loading; it does not by itself show how quickly the model generates once ready.

Tips

Prefer an SSD for model files and system responsiveness, but remember that storage speed does not replace RAM for active inference.
Test with the tasks you actually perform. Code, long documents, and casual chat can favor different model sizes and context settings.
Keep a small note of model name, quantization, runtime version, thread count, context length, and measured speed.
If your runtime offers CPU-specific settings, read its documentation before using them; option names and effects vary.

Troubleshooting

The model is extremely slow or the PC becomes unresponsive: Stop generation, close other applications, and check memory use. Try a smaller model or lower-memory quantization. If the system is swapping heavily, reducing context alone may not be enough.

The runtime will not load the model: Confirm that the file format and model architecture are supported by that runtime, the download completed, and there is sufficient memory. If you selected a CPU instruction-specific build or option, switch to a compatible general build.

Speed gets worse during a long session: Check for CPU temperature and power throttling, background processes, and growing conversation history. Improve airflow, use an appropriate power mode, and start a fresh conversation if old context is unnecessary.

Answers became noticeably worse after changing quantization: Try a higher-bit quantization or a different model. Quantization is a trade-off, and the best choice depends on the model and task.

Thread changes make no difference: Restore automatic settings, verify the runtime is actually using the CPU-optimized build, and check whether memory bandwidth or prompt length is the limiting factor. Not every workload scales with additional threads.

Frequently Asked Questions

Can I make a CPU-only LLM as fast as one using a GPU?
Usually not for comparable models and settings. A CPU can run local models effectively, but a suitable GPU often processes model operations faster.

Does more RAM make inference faster?
More RAM helps when the current setup is short on memory or swapping. Once the model and working data fit comfortably, extra RAM alone may not increase token generation speed.

Is a lower quantization always better for speed?
No. Lower-bit quantization often reduces memory use, but actual speed depends on the model, runtime, processor, and supported operations. It can also reduce answer quality.

Should I use every CPU core?
Not automatically. Start with automatic settings or a count near the physical core count, then benchmark nearby values. The best setting depends on the workload and processor.

Why is the first response slower than later responses?
The runtime may need to load model data, initialize computation, or process a long prompt. Compare repeated runs separately from startup time.

Will a shorter context make the model forget less?
No. A shorter context holds less conversation history, so earlier details may be unavailable. Use the smallest context that still accommodates the prompt and relevant history.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.