Top picks by hardware tier
| GPU tier | Recommended model | Why |
|---|---|---|
| Laptop / single consumer GPU (8-16GB VRAM) | Llama 3 8B, Qwen 2.5 7B, Gemma 9B | Quantized 4-bit fits in consumer GPU memory |
| Single workstation GPU (24GB VRAM) | Llama 70B (4-bit), Qwen 32B, Mistral Small | Mid-size quality on one A6000/RTX 5090-class card |
| Single H100 / MI300 (80GB) | Llama 70B (FP16), DeepSeek V3 distilled | Full-precision mid-large models |
| Multi-GPU server (2-8 GPUs) | Llama 405B, DeepSeek V3, Qwen 72B | Tensor parallel for frontier-scale models |
| Cluster (multi-node) | Llama 405B, DeepSeek V3, Mixtral 8x22B | Full frontier scale; production serving |
The major open-weights families
The most-deployed open-weights family. Strong general capability across sizes from 1B to 405B+ parameters. Llama community license allows commercial use with revenue threshold caveats. Wide ecosystem support — every inference server, every fine-tuning toolkit, every quantization library supports Llama.
Best for: general chat, RAG backends, coding (Llama Coder variants), most production self-hosted workloads.
Strong on coding, math, and reasoning at every size. Qwen Coder variants are competitive with frontier closed models on isolated code tasks. Apache 2.0 license is friendly. Comes in a wide range of sizes (0.5B up to 72B+).
Best for: coding, math, reasoning-heavy workloads, multilingual including Chinese.
Mixture-of-experts design with frontier-grade capability at significantly lower inference cost than dense models of comparable size. MIT license. Requires substantial hardware to run at full precision but quantized variants are tractable on multi-GPU workstations.
Best for: capability-critical workloads where open-weights matters; cost-efficient inference at scale.
Strong quality-per-parameter; Mistral 7B and Mixtral MoE models are efficient inference targets. Apache 2.0 license. The model family that established many of the open-weights conventions (sliding-window attention, GQA, MoE).
Best for: efficient deployment, cost-sensitive workloads, MoE-based architectures.
Google's open-weights family. Smaller sizes (2B, 9B, 27B) optimized for efficient inference. Good safety tuning out of the box. Available with various licenses depending on size.
Best for: small-model deployments, safety-conscious applications.
Hardware sizing rules of thumb
To estimate GPU VRAM needed for a given model:
- FP16 (full precision): ~2 GB per billion parameters. Llama 70B FP16 needs ~140 GB VRAM (multi-GPU).
- INT8: ~1 GB per billion parameters. Llama 70B INT8 fits on a single 80GB card.
- INT4 (4-bit quantization): ~0.5 GB per billion parameters. Llama 70B 4-bit fits on a 40GB card; 7B fits on a 4GB card.
- Add ~20% for KV cache (more for long contexts; see inference latency components).
Picking by use case
- Chat / general assistant: Llama 8B or Qwen 7B on consumer GPU; Llama 70B or Qwen 32B on workstation; DeepSeek V3 or Llama 405B on cluster.
- Coding: Qwen Coder, DeepSeek Coder, Llama Coder. All open-weights coder models are good; pick the size your hardware supports.
- Embedding: different model class — see best embedding model.
- Reasoning: Qwen, DeepSeek-R1 family, Llama with reasoning prompting.
Inference serving
The model is half the story; the inference server is the other half. For production:
- vLLM: dominant open-source inference server. Continuous batching, paged attention, broad model support. See vLLM vs TGI.
- TGI (Text Generation Inference): Hugging Face's server. Similar feature set; tighter HF ecosystem integration.
- Ollama, LM Studio: easier desktop deployment; less production-oriented.
- TensorRT-LLM: NVIDIA-optimized; highest throughput on NVIDIA hardware; more complex setup.
See inference server architecture for the deeper view.
Frequently Asked Questions
What is the best open-source LLM for self-hosting?
Llama is the most-deployed and most-supported. Qwen is the strongest for coding and reasoning. DeepSeek V3 is the most capable open-weights frontier model. The right pick depends on your hardware and workload — not a single "best."
Can I run Llama 70B on a single GPU?
Yes, with 4-bit quantization on a 40-48 GB GPU (RTX 6000 Ada, A6000, A100 40/80GB). Quality is close to FP16 for most tasks. Throughput is lower than full precision but the cost is dramatically lower.
What is the cheapest LLM to self-host?
Small models like Llama 3 8B or Qwen 2.5 7B can run on a single consumer GPU (RTX 4060 or better at 4-bit quantization). For desktop use, no GPU is required for tiny models like Qwen 0.5B running on CPU — slow but functional.
Is self-hosting cheaper than paying for OpenAI / Anthropic API?
It depends on usage. At low volume, API per-call pricing wins because you pay only for what you use. At high steady volume (millions of tokens per day), self-hosted breaks even and beats API pricing. The crossover is workload-dependent but typically falls between $1,000 and $10,000 per month of API spend.
What about licensing?
Apache 2.0 / MIT (Qwen, Mistral, DeepSeek) — full commercial use, minimal restrictions. Llama Community License — commercial use allowed below a revenue threshold (currently 700M monthly active users). Gemma — Google-specific terms. Always check the license for your intended use.