Self-hosted AI has absolutely crossed the threshold from "fun toy" to "actually useful tool" thanks to quantized models and plugins that are surprisingly capable, even with modest hardware. Grabbing an old PC and loading a model grants you your own private AI assistant that doesn't phone home, and that's a powerful thing, but it's not the whole picture.
Running a local LLM taught me a lot about what actually bottlenecks a system, the software trade-offs you'll run into, and how to taper expectations going into a self-hosting project like this. Despite doing my due diligence before jumping in, I still wish someone told me these five things before I started hosting a local LLM.
Your GPU probably isn't the bottleneck you think it is
Your old gaming card is more capable than you realize
When people talk about local LLMs, the conversation almost immediately collapses into VRAM math. How many gigabytes you have determines how big a model you can load, which in turn dictates how useful the model is, but in reality, it's only part of the equation. My home lab setup features a GTX 1070, which isn't an AI beast by any stretch, but it's definitely capable of running modern quantized models. A 7B model at Q4 or Q5 fits comfortably, and even larger variants will load, but their usability is dubious.
On my system, the first real bottleneck wasn’t VRAM at all, but instead, it was memory bandwidth and CPU overhead. Running the model inside a Proxmox VM means the CPU is doing more work than you expect, especially when you’re shuffling tensors between system RAM and VRAM. Disk speed matters too; model load times can feel especially sluggish if you’re pulling multi-gigabyte files from slow storage. This wasn't an issue for me, as everything was running off of a PCIe Gen 3 drive.
Local doesn't always mean private
Quiet cloud dependencies can catch you off
One of the main reasons people self-host is privacy, and the benefits are obvious. No prompts leave your network, no data harvesting, no opaque retention policies, the benefits go on. But that only holds if you’re careful about everything around the model. Many popular local LLM frontends quietly reintroduce cloud dependencies: update checks, embedding APIs, plugin systems, even optional analytics. It’s also really easy to bind a web UI to all interfaces, toss it behind a reverse proxy, and accidentally expose something you meant to keep private, especially if you're new to self-hosting. If your true goal with self-hosting an LLM is privacy, you need to think about where conversations are stored, how long they persist, and who can access the UI.
Prompts matter
Compared to the massive cloud-based models, local stuff needs good prompting
A well-prompted 7B model can feel sharper and more reliable than a poorly prompted 13B model struggling under hardware constraints. System prompts suddenly matter a lot more because there’s no hidden fine-tuning layer compensating for vague instructions, and what you actually type matters too.
When you compare either to a cloud-based solution like ChatGPT or Claude, you can get away with bad prompting with those massive models because it can better infer what you're asking. With these smaller, locally hosted models, being specific is really important.
A local model isn't a 1:1 replacement for a subscription
It's just not as capable, full-stop
Local AI isn't a full replacement for a subscription to ChatGPT or Claude. Even if you're running multiple powerful GPUs in a purpose-built hardware stack with a massive local model, it's still tiny in comparison to what these large companies have built, and that's okay.
Where a local LLM does shine is in the form of an always-available private assistant. It can summarize, search local files, help with scripting, or act as a light sounding board for ideas. It’s fast enough, reliable enough, and completely under your control. Benchmarking it against these massive services is always unfair, as it's just not capable enough to replace your subscriptions.
Local LLMs are very usable, but only in the right context
Self-hosting a local LLM is one of those projects that rewards patience more than raw hardware. If you go in expecting to recreate a polished cloud AI experience, you’ll mostly end up frustrated. If you treat it like any other piece of self-hosted infrastructure, which is something imperfect, configurable, and deeply dependent on how you use it, it starts to make a lot more sense. Even on older hardware, a local model can be genuinely useful when it’s playing to its strengths: privacy, control, and tight integration with your own systems.
