I was never one to fully utilize artificial intelligence with the available tools that we had at the time. Large language models (LLMs) have been around for a good few years now and have refined to the point that they are incredibly useful for handling routine tasks, solving problems, and even holding conversations. I've dabbled in self-hosting my own LLMs instead of using cloud-based solutions and found them to be much more capable when combined with Home Assistant.
LLMs can be useful
So long as you use them properly
An LLM isn't necessarily required by anyone to complete a task. These handy assistants can be used to summarize an email, but why wouldn't you simply read through it? I've seen these prompts miss out important details that I would have otherwise overlooked if relying on an AI summary alone. It also makes me feel more detached from the conversation if I'm not even reading responses. These days, you can even have emails drafted on the fly, so you'll end up watching dialogue in third-person.
Copilot has been ridiculed on Windows for being somewhat useless. It's also not that great to begin with, but Microsoft continues to force it everywhere, and partners are more than happy to tag along for the ride. We saw Adobe and HP glorify the AI push alongside Microsoft, with each company back patting the next as they attempted to convince the CES audience that AI was the future for consumer PCs. AI isn't for everyone, and that's okay — We don't need an NPU on every device.
The same goes for the smart home. LLMs can be useful, but only if you need them. Can you build out a smart home without AI? Absolutely, and I'd go as far as to recommend it, just so it's clear how your smart home is configured. Once solid foundations have been laid, we can then consider adding an LLM to enhance the experience. I did just this for our smart home. AI wasn't even on the cards when I first started dabbing into the Philips Hue ecosystem.
Fast-forward a decade, and now I'm running self-hosted LLMs fully integrated into Home Assistant and other services. Home Assistant by itself is a fairly powerful tool for smart homes. It has everything you'd need to create a smarter abode with integrations for countless brands, support for custom community-made plugins, and even the ability to create and run your own voice assistant. But with the added power of an LLM, it can become quite the hub.
Creating the ultimate assistant
Understands almost any command
It's actually incredibly easy to integrate a locally-hosted LLM into Home Assistant. All you need is a way to record your voice, which can be as simple as a basic USB microphone. I recommend the Home Assistant Voice Preview Edition. Then, install Whisper for speech-to-text and Piper for text-to-speech. With these two plugins installed and configured, you can now select the desired conversation agent within the Home Assistant Voice settings area.
By default, Home Assistant is used here, but we can switch things up by selecting our locally hosted LLM. That's precisely what I did for our installation with the Voice Preview Edition. After this has been linked to your hosted LLM (I highly recommend using Open WebUI and Ollama), the LLM can be used to interpret all the human language recorded by your input device. This makes commands much more user-friendly and natural.
Instead of having to ask Alexa, Google, or even Nabu to turn off a specific light or device, we can ask Home Assistant to switch everything off in a select area or room, leaving one or two devices online. For instance, I could ask the assistant to configure our living room for movie night or simply ask for all lights to be turned off aside from our free-standing lamp to create a more relaxed ambience. It can understand our commands and carry them out quickly.
The best part? Your data stays local, and there are no hidden fees for the luxury.
How LLMs changed our home
Before making use of locally hosted LLMs, I would throw a message or two to hosted AI chatbots, such as ChatGPT. These were fine for running through checks or asking for lighter assistance, but you can't really do much outside of predetermined boundaries. These LLMs are hosted on company infrastructure and can't really be integrated into your self-hosted empire without forking out some money, but even then, you're relying on an external connection.
Prior to this, I would configure and run numerous automations. Interacting with Home Assistant through voice would be better than Alexa and Google, but it would still feel slightly rigid and cumbersome. Without the exact name of a device or phrase, many conversations with a smart assistant would end up with polite declines since they wouldn't be able to parse what I desired. Running an LLM and pairing it up with Home Assistant allowed me to use intent-based requests.
LLMs can be used elsewhere within Home Assistant, too. Have you everwondered why an automation didn't run successfully? LLMs can analyze and explain why something works, or better still, why it didn't work. We can ask, "Why did the garage lights and camera trigger last night?" The LLM could then check everything over and relay reasoning as to why these devices activated.
They really have transformed our smart home, and it was much easier to integrate everything than we initially thought. And if you're distrustful of current cloud-based home assistants, an LLM, Home Assistant, and some free time may be all that's required to start your self-hosted journey. You could even go with a GLaDOS-powered voice assistant, because why not?
