Best Local AI Model for Home Assistant Isn’t Always the Biggest
The best local AI model for Home Assistant right now is Qwen 2.5 3B (via Ollama) for most users. It strikes the sweet spot between fast response times on modest hardware (like an 8 GB Raspberry Pi 5 or a small NUC) and enough intelligence to handle voice commands, intent parsing, and basic automation suggestions without the lag of larger models.
What to Look For
When choosing a local AI model for Home Assistant, the biggest model isn’t automatically the best. Here are the criteria that actually matter:
- RAM and VRAM footprint: The model must fit in your available memory (ideally with room for context). A 7B model quantized to Q4 needs ~4-5 GB; a 13B needs ~8-10 GB; a 70B needs 40+ GB. If it doesn’t fit, it’ll swap and become unusably slow.
- Inference speed: For voice assistants (like Assist), you want responses in under 2-3 seconds. That means tokens per second (t/s) matters more than raw accuracy. Smaller models often win here.
- Tool calling / function calling: Home Assistant’s AI needs to call services (turn on lights, set thermostats). Models that support structured tool calling (like Qwen, Llama 3.1, and Mistral) integrate more reliably than those that don’t.
- Context window: You need enough context to hold your conversation history and the current state of your devices. 8K is a bare minimum; 32K is comfortable.
- Ease of setup: Ollama and the Home Assistant Ollama integration make this a one-command install. Models that require custom scripts or conversion are a hassle to maintain.
Qwen 2.5 3B (via Ollama)
Strengths: Very fast on CPU-only hardware (10-20 t/s on a Raspberry Pi 5), supports function calling natively, and has a 32K context window. It’s also small enough to run alongside Home Assistant on the same device without starving other add-ons.
Trade-offs: It’s not great at complex reasoning or long multi-step instructions. It can occasionally misinterpret ambiguous commands.
Who it suits: Most Home Assistant users with a Raspberry Pi 4/5, a small NUC, or an older laptop running HAOS. If you want a snappy voice assistant without buying new hardware, this is your pick.
Llama 3.2 3B (via Ollama)
Strengths: Meta’s latest small model, with strong instruction following and good tool calling. It’s also well-optimized for Apple Silicon and runs fast on M1/M2 Macs.
Trade-offs: Slightly less capable at multilingual tasks than Qwen. Context window is 128K, but that can slow things down if you actually use it all.
Who it suits: Users on Apple Silicon Macs or any system where you want a reliable, well-supported small model. It’s a great alternative if Qwen doesn’t behave well with your setup.
Mistral 7B (via Ollama)
Strengths: A 7B model that’s still fast on modest hardware (5-10 t/s on a Pi 5 with 8GB RAM). It has excellent natural language understanding and decent tool calling. It’s also one of the most widely tested models in Home Assistant communities.
Trade-offs: Needs ~5 GB RAM, so it won’t fit on a 4 GB Pi. It’s slower than 3B models, so voice responses may have a noticeable delay.
Who it suits: Users with a bit more RAM (8 GB+ on the HA device) who want better comprehension without jumping to a huge model. Good for text-based automations where latency is less critical.
Llama 3.1 8B (via Ollama)
Strengths: A step up in intelligence from 3B models, with strong reasoning and tool calling. Runs well on a 16 GB NUC or a desktop with a decent CPU (10+ t/s).
Trade-offs: Requires ~6 GB RAM, so it’s not for small Pis. It’s also slower than 3B models, so voice assistants may feel sluggish.
Who it suits: Users with a dedicated HA server that has 16 GB RAM or more, and who want a model that can handle more complex queries and multi-step automation suggestions.
Qwen 2.5 7B (via Ollama)
Strengths: Better multilingual support and stronger tool calling than the 3B version. It’s a good middle ground between speed and capability.
Trade-offs: Same RAM requirements as Mistral 7B (~5 GB). It’s slower than 3B, so not ideal for voice if you’re on a Pi.
Who it suits: Users who need better language coverage (e.g., non-English) and have a bit more hardware headroom. It’s a solid choice for text-based automation and chat.
Phi-3 Mini 3.8B (via Ollama)
Strengths: Microsoft’s small model is surprisingly good at reasoning for its size, and it’s very fast on CPU (similar to Qwen 3B). It also has a 128K context window.
Trade-offs: Tool calling is less reliable than Qwen or Llama. It’s also less common in Home Assistant tutorials, so you may need to troubleshoot more.
Who it suits: Tinkerers who want to experiment with a different architecture and don’t rely heavily on function calling. Good for simple chat and basic automations.
How to Choose
Start with your hardware. If you have a Raspberry Pi 4 or 5 with 4-8 GB RAM, stick to 3B models (Qwen 2.5 3B or Llama 3.2 3B). If you have 16 GB RAM on a NUC or desktop, try a 7B/8B model. If you have a dedicated GPU with 8+ GB VRAM, you can run 13B models like Llama 3.1 8B or even a 14B model, but that’s overkill for most home automation.
Next, think about your primary use case. For voice control via Assist, prioritize speed—a 3B model will feel much more responsive. For complex automations where you ask the AI to write scripts or debug configurations, a larger model (7B+) will understand better.
Finally, consider the trade-off between speed and intelligence. A 3B model might occasionally misinterpret a command, but it’s fast enough that you can just repeat yourself. A 7B model is smarter but adds a second or two of latency. There’s no right answer—it depends on your patience and your hardware.
A practical approach: install Ollama, pull Qwen 2.5 3B, and test it for a week. If you find it too dumb, try Llama 3.1 8B. If that’s too slow, go back. The beauty of local models is you can switch in minutes.
Frequently Asked Questions
Can I run a local AI model on a Raspberry Pi 4?
Yes, but stick to 3B models. Qwen 2.5 3B runs fine on a Pi 4 with 4 GB RAM, though it may be a bit slower than on a Pi 5. Avoid 7B models on a Pi 4—they’ll be painfully slow.
Do I need a GPU to run these models?
No, all the models listed run on CPU. A GPU speeds things up, but you can get acceptable performance on a modern CPU with 8+ GB RAM. For a Pi, CPU-only is the norm.
How do I integrate a local model with Home Assistant?
Use the Ollama add-on in Home Assistant OS or Supervised, then configure the Ollama integration with the model name. The official Home Assistant documentation has step-by-step instructions.
Will a larger model like Llama 3 70B work?
Only if you have a very powerful machine (e.g., a server with 64+ GB RAM or a high-end GPU). For most home setups, it’s overkill and will be too slow for voice. Stick to 3B-8B.
What about models like GPT-4 or Claude?
Those are cloud-based, not local. Local models are privacy-friendly and free to run, but they’re less capable. If you’re okay with sending data to the cloud, you can use those, but this guide is about local options.
Can I use a model for more than just voice? Like for automations?
Yes, you can use the AI to generate automation scripts or explain errors. The Ollama integration exposes a conversation agent that can be used in automations. Larger models are better at this, but even 3B can handle simple tasks.
Bottom Line
For most Home Assistant users, Qwen 2.5 3B is the best local AI model because it’s fast, small, and supports tool calling. If you have more RAM and want smarter responses, step up to Llama 3.1 8B. Start with the 3B, test it, and upgrade only if you find it lacking. The biggest model is rarely the best choice when you have to wait for it.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.