Self-Hosted AI: Hardware Requirements & Best Models (2026)

Running AI models on your own infrastructure instead of calling cloud APIs gives you three things that no hosted service can: complete data privacy, predictable costs, and the freedom to choose any model. The trade-off is that you need the right hardware and a basic understanding of how large language models use memory.

This guide covers the server side of self-hosting: what hardware you actually need, which open-weight models are worth running in late 2026, how to size a machine that a whole team shares, and how the costs compare to cloud APIs. If you want a hands-on tutorial with Docker Compose and DeployHQ, see our step-by-step guide to self-hosting Open WebUI and Ollama on a VPS. If you're trying to run a coding model on your own laptop or workstation rather than a server, our guide to running AI coding models locally is the better fit.

Why self-host AI models?

Data stays on your servers

When you call the OpenAI or Anthropic API, every prompt and response passes through their servers. For most use cases that is fine — but if you work with customer PII, medical records, legal documents, or proprietary code, sending that data to a third party may violate compliance requirements or internal security policies.

Self-hosted models process everything locally. The data never leaves your network.

Predictable, fixed costs

Cloud API pricing scales with every token. A self-hosted model costs the same whether it answers ten requests a day or ten thousand — you pay for the server, not the usage. The catch is that a server has a throughput ceiling (more on that in the team sizing section below), so fixed cost does not mean unlimited capacity.

Full control over models and behaviour

You choose which model to run, how it is configured, and what system prompts it uses. You can fine-tune models on your own data, swap models without changing application code, and run multiple models side-by-side for different tasks.

What a self-hosted AI stack looks like

Self-hosted AI usually means more than one component. A typical production setup has four layers:

You can start with just the first two. Our foundational guide to the Ollama + Open WebUI stack explains how the runtime, the UI and the Nginx reverse proxy fit together. If you want automations on top, n8n maintains a Self-hosted AI Starter Kit that bundles n8n with Ollama, Qdrant and PostgreSQL — and we have a guide to self-hosting n8n on a VPS with Docker.

Understanding the hardware: why memory matters most

The single most important factor in self-hosting AI is memory — GPU VRAM if you have a GPU, system RAM if you don't. A language model must be loaded into memory before it can generate text. If it fits entirely in VRAM, inference is fast. If it spills over into system RAM, generation slows dramatically, because every generated token has to read the model's weights from much slower memory.

Total memory needed = model weights + KV cache + runtime overhead. Weights are fixed by the model and its quantisation. The KV cache grows with context length and with the number of requests you serve at once — which is why a model that runs fine for one person can fall over when five colleagues use it.

Quantisation: the key to fitting models in less memory

Full-precision models use 16 bits per parameter (FP16/BF16). Quantisation stores the weights at lower precision so they take less memory. The llama.cpp project publishes measured sizes for each format; here they are for Llama 3.1 8B:

Source: llama.cpp quantize README.

Q4_K_M is the common choice for self-hosting — it is, for example, the quantisation of Ollama's qwen3.8:27b tag. As a rule of thumb, budget roughly 0.6 GB per billion parameters at 4-bit, before KV cache and overhead. At that rate a 70B dense model needs around 43 GB for the weights alone, which is why it doesn't fit on a single 24 GB or even 32 GB card.

Some newer models ship pre-quantised: OpenAI's gpt-oss models are post-trained with their mixture-of-experts weights in MXFP4 (4.25 bits per parameter), which is why gpt-oss:20b is a 14 GB download.

Dense vs mixture-of-experts (MoE): memory vs speed

Many current open models are mixture-of-experts: only a fraction of the parameters (the active parameters) are used for each token. This splits the sizing question in two:

• Memory is sized by total parameters. All experts must be loaded, so a 120B MoE needs as much memory as a 120B dense model at the same quantisation.
• Speed tracks active parameters. Each token only touches the active experts, so a model like gpt-oss-120b (117B total, 5.1B active) generates far faster than a 120B dense model would on the same hardware.

That makes small-active-parameter MoE models the best fit for CPU-only servers, where memory is plentiful but compute and memory bandwidth are limited.

Hardware tiers and what fits

The download sizes below come from the official Ollama library tags (default quantisation). Leave headroom on top for the KV cache.

Not sure if your hardware is enough? Can I Run AI? estimates whether a specific model fits your machine before you download anything.

CPU-only is viable for small models and background work. Generation on CPU is much slower than on a GPU, which makes it a poor fit for interactive chat with many users, but it works for batch summarisation, classification, or overnight document processing where latency doesn't matter.

Already have a server picked out? You can wire up its config repo to deploy from GitHub to your server automatically, so model and Docker changes ship on every push instead of over SSH.

Best open-weight models for self-hosting in 2026

The open model landscape moves fast — several of the models below were released in the last six months. Rather than chase leaderboards, pick by the memory you have and the licence you can accept.

Models for a single server

For coding-specific models and how to wire them into Cline, Aider or Continue, see the local coding-models guide linked at the top of this article rather than duplicating it here.

Frontier open-weight models (multi-GPU only)

These models compete with the top hosted APIs, but they need a multi-GPU node and a production-grade runtime (vLLM or SGLang) — they are realistic only for organisations with dedicated ML infrastructure:

Even at 4-bit, the 0.6 GB-per-billion rule puts DeepSeek-V4-Flash's weights well above 150 GB — several data-centre GPUs before you add any KV cache. If you want to experiment with DeepSeek on a single VPS, the smaller distilled models are the practical route; see our guide to running DeepSeek on a VPS with Ollama.

Licence gotchas to check before you deploy

Open weights does not always mean use it however you like. For a company deployment, read the licence file in each model repo:

• Apache 2.0 / MIT (gpt-oss, Gemma 4, Qwen3.5/3.6, Qwen3.8-27B, DeepSeek V4, GLM-5.2) — permissive. Note that Gemma 4 is Apache 2.0, whereas Gemma 3 shipped under Google's separate Gemma terms.
• Kimi K2.6 (Modified MIT) — if you use it in a commercial product with more than 100 million monthly active users or more than US$20 million in monthly revenue, you must prominently display Kimi K2.6 in the UI.
• GLM-5.3 — unlike GLM-5.2, it uses a custom licence: a Model as a Service operator whose group revenue exceeds US$10 billion over 12 months must pass Z.AI's security review before commercial use.
• Licences can differ within one family. Qwen3.8-27B is Apache 2.0, but the flagship Qwen3.8-2.4T-A95B uses a separate qwen3.8-max licence. Llama 4 uses Meta's Llama 4 community licence and is gated on Hugging Face.
• The interface has a licence too. Open WebUI's licence forbids removing or replacing its branding in deployments with more than 50 end users in any rolling 30-day period, unless you have permission or an enterprise licence. White-labelling it for a large team or a client is not free.

Runtime tools for self-hosting

You do not interact with model weights directly. A runtime loads the model and serves an API. The main options for servers:

Hugging Face TGI is no longer a good choice for new deployments. The project is in maintenance mode and its repository was archived in March 2026; the maintainers point users to vLLM, SGLang, llama.cpp or MLX instead.

Ollama is the easiest starting point. On Linux it installs with a single command:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull gpt-oss:20b

vLLM is what you graduate to when many users hit the same model. Serving a model from Hugging Face is one command:

vllm serve Qwen/Qwen3.8-27B

llama.cpp can pull a GGUF directly from Hugging Face and expose an OpenAI-compatible server:

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Combined with Open WebUI, any of these provides a full ChatGPT-like interface.

Sizing a shared team server: concurrency and context

This is where most self-hosted setups that worked on my machine break. Three Ollama defaults matter as soon as more than one person uses the server (from the Ollama FAQ and its context-length docs):

• The default context window depends on your VRAM. Ollama's docs set it to 4k under 24 GiB of VRAM, 32k at 24–48 GiB and 256k at 48 GiB or more (some pages of the FAQ still quote 4,096). Long documents and big code files get truncated at the default unless you raise it with OLLAMA_CONTEXT_LENGTH, even when the model itself supports 128K or 256K. Set it explicitly and check what you got with ollama ps; on a large card, a 256k default multiplies the KV cache for every parallel slot.
• One request at a time per model by default. OLLAMA_NUM_PARALLEL defaults to 1; extra requests wait in a queue (OLLAMA_MAX_QUEUE, default 512).
• Memory scales with both. Ollama's docs state that required RAM scales by OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH. Raising parallelism to 4 with a 32K context reserves KV cache for 128K tokens.

Models also stay loaded for 5 minutes after the last request by default, so the first request after an idle period pays the load time.

Two levers recover memory:

1. Quantise the KV cache. Set OLLAMA_KV_CACHE_TYPE=q8_0 (roughly half the memory of the f16 default, with very small precision loss per Ollama's docs) or q4_0 (roughly a quarter, with a more noticeable loss at long context). This requires Flash Attention, which Ollama enables automatically on supported hardware.
2. Pick an MoE model. Fewer active parameters means each request finishes faster, so the queue drains quicker on the same hardware.

If you regularly have several people waiting in Ollama's queue, that's the signal to move to vLLM or SGLang, which batch concurrent requests together. vLLM by default claims most of the GPU's memory for weights plus KV cache (--gpu-memory-utilization), takes the context length from the model config unless you set --max-model-len, and splits a model across GPUs with --tensor-parallel-size.

Cost comparison: self-hosted vs. cloud APIs

Exact numbers go stale every time a provider changes its price list, so run the comparison with your own figures:

• Cloud API cost per month = (input tokens × input price) + (output tokens × output price). It scales linearly with usage and is zero when nobody is using it.
• Self-hosted cost per month = server rent (or amortised hardware + power) + the engineering time to patch, monitor and upgrade it. It's flat whether the server is busy or idle.

Things that make the comparison honest:

• Compare like with like. A 20–30B open model is not a substitute for the largest hosted models on hard reasoning tasks. Compare against the hosted model tier you'd actually replace.
• Capacity is the ceiling, not tokens. A single GPU serves a limited number of requests at once (see above). If peak usage needs three GPUs, you pay for three GPUs all month.
• Idle hardware still costs money. Self-hosting wins when utilisation is steady and high; spiky or low usage favours pay-per-token APIs.
• Privacy can decide it outright. If the data can't leave your network, the cost comparison is secondary.

For teams that need both privacy and top-tier quality, a hybrid approach works well: run a local model for routine tasks (summarisation, drafting, internal Q&A) and route only complex requests to a cloud API. Open WebUI supports both local Ollama models and cloud API keys in the same interface. For more control, a gateway gives you one endpoint for both — compare the options in our roundup of AI model routers including the self-hostable LiteLLM, or see how a hosted router works in our OpenRouter guide for teams.

Deploying and managing self-hosted AI with DeployHQ

Once your AI stack is running, you need a way to manage configuration changes, model updates, and Nginx rules without SSH-ing into the server every time.

DeployHQ automates this by deploying from a Git repository to your VPS via SSH. Keep your docker-compose.yml, Nginx config and a list of models in a repo, push a change, and DeployHQ handles the rest.

Key DeployHQ features for AI deployments:

• SSH commands run after each deploy — restart Docker containers, ollama pull any new models in your list
• Config files inject .env secrets without committing them to Git
• Build pipelines run build or validation steps before deploying
• Automatic deployments on every push to your main branch
• One-click rollback if a model swap or config change misbehaves

If you're new to containers, start with our explainer on what Docker is and how images and containers work. Ready to automate your AI server? Start a free DeployHQ trial.

Security best practices

Self-hosting gives you control, but also responsibility:

• Network isolation: Ollama binds to 127.0.0.1:11434 by default — keep it that way, and never expose Ollama or vLLM directly to the internet. Model APIs have no user accounts of their own.
• Watch Docker port publishing: the standard docker run -p 11434:11434 ollama/ollama publishes the port on all interfaces, and Docker's documentation notes that published container ports bypass ufw rules. Publish to localhost only (-p 127.0.0.1:11434:11434) and let the reverse proxy be the only public entry point.
• Reverse proxy with TLS: use Nginx or Caddy to terminate HTTPS in front of your interface — our Caddy vs Nginx vs Apache comparison covers the trade-offs
• Access control: Open WebUI supports user accounts with role-based access; disable public signup
• Harden the host: patch the OS, use SSH keys only, and firewall everything except SSH and HTTPS — see our Linux server hardening checklist
• Update regularly: model runtimes (Ollama, vLLM, llama.cpp) ship frequent releases, including security fixes
• Monitor resource usage: a runaway request or a too-high parallelism setting can exhaust memory; set memory limits in Docker
• Protect API keys: if bridging to cloud APIs, follow our guide to keeping API keys out of your Git repository, never hardcode keys

Related guides

• How to Deploy Your First AI Agent to a VPS with Docker — package and ship an agent that calls your self-hosted model

If you have questions or need help, reach out at support@deployhq.com or on X (@deployhq).