VRAM is the deciding spec
A model has to fit in GPU memory to run well. If it doesn't fit, it spills to system RAM or CPU and slows to a crawl. So pick the GPU by how much VRAM you need for your target model, then worry about speed. Rough 4-bit quantized requirements:
| VRAM | LLM size (4-bit) | Also good for |
|---|---|---|
| 8 GB | ~7–8B | Image generation (SD/Flux), small LLMs |
| 12 GB | ~13B | Comfortable image gen, mid-small LLMs |
| 16 GB | ~22–32B (tight) | Most hobbyist workloads |
| 24 GB | ~32B comfortably | Serious local AI, larger image models |
| 48 GB | ~70B (4-bit) | Workstation-grade local inference |
| 2× 24 GB or 80 GB | 70B FP16 / larger | Multi-GPU and frontier models |
Best GPUs by tier
The sweet spot for serious local AI. 24 GB runs 32B models comfortably at 4-bit, handles every image model with room to spare, and offers strong compute for fast generation. The best single-card choice for most enthusiasts who want headroom without workstation prices.
Best for: 32B LLMs, fast image generation, future-proofing a single-GPU build.
The practical entry point. 16 GB runs 7–13B models comfortably and squeezes into the low-20Bs with aggressive quantization, while handling Stable Diffusion and Flux well. The best balance of price and capability for getting started with local AI.
Best for: first local-AI build, 7–13B LLMs, image generation on a budget.
When you need 70B models on a single card. 48 GB fits 70B at 4-bit with context headroom and runs large image and video models. Pricey and aimed at professionals, but the simplest path to large-model inference without multi-GPU complexity.
Best for: 70B local inference, professional workloads, single-card simplicity.
Macs with large unified memory (32–192 GB) can run surprisingly big models because CPU and GPU share one memory pool. Slower than a high-end NVIDIA card for raw throughput, but the high memory ceiling lets a well-specced Mac run models that would need multiple PC GPUs.
Best for: Mac users, large models on a quiet desktop, memory over raw speed.
For 70B at full precision or frontier MoE models, you need 80 GB data-center cards or multiple GPUs with tensor parallelism. This is production territory — expensive and complex, but the only way to serve the largest open-weights models locally at speed.
Best for: production serving, frontier models, teams running heavy inference.
Why NVIDIA still dominates
CUDA remains the default target for AI tooling, so NVIDIA cards have the smoothest software experience across inference servers, fine-tuning libraries, and image-generation tools. AMD (ROCm) and Apple (Metal/MLX) have improved a lot and work well for many tasks, but for the widest compatibility and least friction, NVIDIA is still the safe choice.
Estimating VRAM for a model
- FP16: ~2 GB per billion parameters.
- INT8: ~1 GB per billion parameters.
- INT4 (4-bit): ~0.5 GB per billion parameters.
- Add ~20% for the KV cache, more for long contexts. See inference latency components.
Don't forget the model download
Model weights are large — multi-gigabyte per model, and you'll often pull several. A fast, stable connection makes setup far less painful, even though generation itself is GPU-bound. Run a speed test to confirm your download throughput before grabbing a library of models.
Frequently Asked Questions
What is the best GPU for local AI in 2026?
A 24 GB consumer flagship is the value sweet spot — it runs 32B LLMs at 4-bit and every image model fast. Drop to a 16 GB card to start on a budget, or step up to 48 GB workstation cards for 70B models on one GPU.
How much VRAM do I need?
It depends on model size: ~8 GB for 7B, ~16 GB for 13B, ~24 GB for 32B, and ~48 GB for 70B at 4-bit quantization. Image models like Stable Diffusion and Flux run comfortably from 8–12 GB.
Is NVIDIA necessary, or can I use AMD or Apple?
NVIDIA has the smoothest software support via CUDA. AMD (ROCm) and Apple Silicon (MLX) work well for many tasks now, and Apple's large unified memory can run big models. NVIDIA remains the lowest-friction choice.
Can I run AI without a dedicated GPU?
Yes, for small models on CPU or Apple Silicon, but it's slow. For responsive LLMs above a few billion parameters or any serious image generation, a dedicated GPU with adequate VRAM makes a huge difference.
Is local AI cheaper than cloud GPUs or APIs?
At high steady usage, owning a GPU beats renting cloud GPUs or paying per-token APIs. At low or bursty usage, cloud and APIs are cheaper because you avoid the upfront hardware cost. The crossover depends on your volume.