How much VRAM do I need to run AI locally?

It depends on the model size and quantization. A 7B model at Q4 needs ~4-5GB, fitting on 8GB GPUs. A 70B model at Q4 needs ~38GB. For most users, 12-16GB VRAM handles 7B-14B models well — that's the sweet spot for local AI in 2026. The RTX 5090 (32GB) or M5 Max (64GB) open up 32B+ models.

What is quantization?

Quantization reduces model precision from 16-bit to lower bit widths (Q4, Q5, Q8), shrinking VRAM requirements 2-4×. Q4_K_M is the recommended balance — it scores ~88% on quality benchmarks while using ¼ the memory. Most users won't notice quality differences at Q4 or above. Q8 (99% quality) is ideal if your GPU has headroom.

Can I run AI on a Mac?

Yes. Apple Silicon Macs (M1-M5) use unified memory shared between CPU and GPU. An M4 Pro (24GB) runs 14B models easily. M5 Max (64GB) handles 32B+ models. The new M5 Ultra (256GB) can even run 70B at full precision. Use Ollama or LM Studio for the easiest setup.

What is CPU offloading?

When a model doesn't fit entirely in VRAM, some layers run on your CPU using system RAM. This works but is 5-10× slower than full GPU inference. A model showing 'offload' status will run, but expect 1-5 tokens/sec instead of 30-60. AMD's Ryzen AI MAX (128GB shared memory) and Apple Ultra chips minimize this issue.

What's the best GPU for running AI locally in 2026?

Best value: RTX 5060 Ti 16GB or RTX 4060 Ti 16GB. Best performance: RTX 5090 (32GB) or RTX 4090 (24GB). Best for huge models: Apple M5 Ultra (256GB) or AMD Ryzen AI MAX+ 395 (128GB shared). VRAM capacity matters more than raw compute speed for running larger models.

How fast will models run on my GPU?

Speed depends on memory bandwidth. An RTX 5090 achieves ~50-100 tokens/sec on 7B models. An RTX 4060 gets ~15-30 t/s. Apple M4 Max gets ~20-40 t/s. Human reading speed is ~4-5 t/s, so even budget GPUs are usable for interactive chat.

What is tokens per second?

Tokens per second (t/s) measures AI text generation speed. 1 token ≈ ¾ of a word. 20+ t/s feels instant, 10-20 t/s is comfortable, 5-10 t/s is usable, below 5 t/s feels slow. ChatGPT streams at 30-60 t/s for reference.

Should I run AI locally or use an API?

Run locally if: you want privacy, unlimited usage, or have a good GPU. Use API if: you need frontier models (GPT-4.1, Claude Opus 4.6, Gemini), want zero setup, or your GPU is too small. Many users do both — local for daily tasks (Qwen 3.5, Gemma 4), API for complex work.

What models are best for coding locally?

DeepSeek-R1 32B and Qwen 3.5 32B lead for coding at the 32B tier (needs 16-24GB). At 14B: Phi-4 and Qwen 3.5 14B are excellent. For 8GB GPUs: Qwen 3.5 7B or DeepSeek-R1 8B at Q4 quantization offer surprisingly good code assistance.

Can My PC Run This AI Model?

Check which AI models your GPU can run locally. Enter your hardware specs and see which LLMs fit in VRAM, with speed estimates and quantization recommendations.

✅ Updated April 2026· 53 models · 73 GPUs

🖥️ Select Your GPU

🟢

RTX 4060

NVIDIA · Ada Lovelace

GB VRAM

272

GB/s

⚙️ System Configuration

System RAM (GB)

Used for CPU offloading when model exceeds VRAM

Context Length (tokens)

Longer context = more VRAM for KV cache

Models Run

Need Offload

Won't Run

🏆 Best Model for Your GPU

Phi-4 14B

✅ Runs GreatQ3_K_M · 6.7GB / 8GB· ~21 tok/s

Coding, STEM · 14B parameters · Phi

Filter:Show only compatible53 models

Full Compatibility Results — RTX 4060

Model	Status	VRAM	Quant	Quality	Speed
Llama 3.2 3B3B Lightweight, edge devices	✅ Runs Great	6.5GB	FP16	🟢 Excellent	~22 t/s 🚀 Fast
Llama 3.2 1B1B Ultra-lightweight, mobile	✅ Runs Great	2.5GB	FP16	🟢 Excellent	~65 t/s ⚡ Blazing fast
Qwen 3 1.7B1.7B Edge / mobile	✅ Runs Great	3.9GB	FP16	🟢 Excellent	~38 t/s 🚀 Fast
Qwen 3 0.6B0.6B Ultra-small, IoT	✅ Runs Great	1.7GB	FP16	🟢 Excellent	~109 t/s ⚡ Blazing fast
Gemma 4 2B2B Edge devices	✅ Runs Great	4.5GB	FP16	🟢 Excellent	~33 t/s 🚀 Fast
Gemma 3 1B1B Tiny, embedded	✅ Runs Great	2.5GB	FP16	🟢 Excellent	~65 t/s ⚡ Blazing fast
DeepSeek-R1 1.5B (Distill)1.5B Ultra-light reasoning	✅ Runs Great	3.5GB	FP16	🟢 Excellent	~44 t/s ⚡ Blazing fast
Falcon 3 3B3B Lightweight	✅ Runs Great	6.5GB	FP16	🟢 Excellent	~22 t/s 🚀 Fast
Falcon 3 1B1B Edge, mobile	✅ Runs Great	2.5GB	FP16	🟢 Excellent	~65 t/s ⚡ Blazing fast
Qwen 3 4B4B Lightweight, fast	✅ Runs Great	4.5GB	Q8_0	🟢 Excellent	~33 t/s 🚀 Fast
Gemma 4 4B4B Fast, efficient	✅ Runs Great	4.5GB	Q8_0	🟢 Excellent	~33 t/s 🚀 Fast
Gemma 3 4B4B Fast, lightweight	✅ Runs Great	4.5GB	Q8_0	🟢 Excellent	~33 t/s 🚀 Fast
Phi-4 Mini 3.8B3.8B Ultra-compact, on-device	✅ Runs Great	4.3GB	Q8_0	🟢 Excellent	~34 t/s 🚀 Fast
Yi 1.5 9B9B Fast bilingual	✅ Runs Great	6.7GB	Q5_K_M	🟢 Excellent	~21 t/s 🚀 Fast
Qwen 3.5 14B14B Balanced quality	✅ Runs Great	6.7GB	Q3_K_M	🟡 Acceptable	~21 t/s 🚀 Fast
Qwen 3 14B14B Balanced performance	✅ Runs Great	6.7GB	Q3_K_M	🟡 Acceptable	~21 t/s 🚀 Fast
DeepSeek-R1 14B (Distill)14B Balanced reasoning	✅ Runs Great	6.7GB	Q3_K_M	🟡 Acceptable	~21 t/s 🚀 Fast
Phi-4 14B14B Coding, STEM	✅ Runs Great	6.7GB	Q3_K_M	🟡 Acceptable	~21 t/s 🚀 Fast
Qwen 3.5 7B7B Fast general use	⚡ Tight Fit	7.6GB	Q8_0	🟢 Excellent	~19 t/s ✅ Comfortable
Mistral 7B v0.37B Fast, efficient	⚡ Tight Fit	7.6GB	Q8_0	🟢 Excellent	~19 t/s ✅ Comfortable
Command R7B7B Lightweight RAG	⚡ Tight Fit	7.6GB	Q8_0	🟢 Excellent	~19 t/s ✅ Comfortable
Llama 3.1 8B8B Fast, consumer GPUs	⚡ Tight Fit	7.0GB	Q6_K	🟢 Excellent	~20 t/s 🚀 Fast
Qwen 3 8B8B Fast general use	⚡ Tight Fit	7.0GB	Q6_K	🟢 Excellent	~20 t/s 🚀 Fast
DeepSeek-R1 8B (Distill)8B Fast reasoning	⚡ Tight Fit	7.0GB	Q6_K	🟢 Excellent	~20 t/s 🚀 Fast
InternLM 3 8B8B Reasoning, tool use	⚡ Tight Fit	7.0GB	Q6_K	🟢 Excellent	~20 t/s 🚀 Fast
Falcon 3 10B10B Multilingual, efficient	⚡ Tight Fit	7.5GB	Q5_K_M	🟢 Excellent	~19 t/s ✅ Comfortable
Gemma 4 12B12B Balanced quality	⚡ Tight Fit	7.3GB	Q4_K_M	🟢 Good	~19 t/s ✅ Comfortable
Gemma 3 12B12B Balanced quality	⚡ Tight Fit	7.3GB	Q4_K_M	🟢 Good	~19 t/s ✅ Comfortable
Mistral Nemo 12B12B Balanced quality	⚡ Tight Fit	7.3GB	Q4_K_M	🟢 Good	~19 t/s ✅ Comfortable
Qwen 3.5 32B32B Coding, math	🐌 CPU Offload	10.8GB	Q2_K	🟡 Acceptable	~1 t/s 🐌 Slow
Qwen 3 32B32B High quality, coding	🐌 CPU Offload	10.8GB	Q2_K	🟡 Acceptable	~1 t/s 🐌 Slow
Gemma 4 27B27B Reasoning, code	🐌 CPU Offload	9.1GB	Q2_K	🟡 Acceptable	~1 t/s 🐌 Slow
Gemma 3 27B27B Code, reasoning	🐌 CPU Offload	9.1GB	Q2_K	🟡 Acceptable	~1 t/s 🐌 Slow
DeepSeek-R1 32B (Distill)32B Reasoning, coding	🐌 CPU Offload	10.8GB	Q2_K	🟡 Acceptable	~1 t/s 🐌 Slow
Mistral Small 3.1 24B24B Fast, function-calling	🐌 CPU Offload	8.1GB	Q2_K	🟡 Acceptable	~1 t/s 🐌 Slow
Mixtral 8x7B46.7B MoE efficiency	🐌 CPU Offload	15.8GB	Q2_K	🟡 Acceptable	~1 t/s 🐌 Slow
Command R 35B35B RAG, enterprise	🐌 CPU Offload	11.9GB	Q2_K	🟡 Acceptable	~1 t/s 🐌 Slow
Yi 1.5 34B34B Bilingual EN/CN	🐌 CPU Offload	11.5GB	Q2_K	🟡 Acceptable	~1 t/s 🐌 Slow
Llama 4 Maverick400B Frontier quality, multimodal	❌ Won't Run	135.5GB	Q2_K	🟡 Acceptable	—
Llama 4 Scout109B Efficient long-context	❌ Won't Run	36.9GB	Q2_K	🟡 Acceptable	—
Llama 3.3 70B70B Best open-source dense	❌ Won't Run	23.7GB	Q2_K	🟡 Acceptable	—
Llama 3.1 405B405B Max quality (multi-GPU)	❌ Won't Run	137.2GB	Q2_K	🟡 Acceptable	—
Llama 3.1 70B70B Strong all-rounder	❌ Won't Run	23.7GB	Q2_K	🟡 Acceptable	—
Qwen 3.5 72B72B Top-tier reasoning	❌ Won't Run	24.4GB	Q2_K	🟡 Acceptable	—
Qwen 3 235B-A22B235B Frontier MoE	❌ Won't Run	79.6GB	Q2_K	🟡 Acceptable	—
Qwen 3 72B72B Top-tier multilingual	❌ Won't Run	24.4GB	Q2_K	🟡 Acceptable	—
DeepSeek-R1 671B671B Frontier reasoning (multi-GPU)	❌ Won't Run	227.3GB	Q2_K	🟡 Acceptable	—
DeepSeek-R1 70B (Distill)70B Strong reasoning	❌ Won't Run	23.7GB	Q2_K	🟡 Acceptable	—
DeepSeek-V3 671B671B General (multi-GPU)	❌ Won't Run	227.3GB	Q2_K	🟡 Acceptable	—
Mistral Large 2 123B123B Frontier, multilingual	❌ Won't Run	41.7GB	Q2_K	🟡 Acceptable	—
Mixtral 8x22B141B MoE frontier	❌ Won't Run	47.8GB	Q2_K	🟡 Acceptable	—
Command R+ 104B104B Enterprise, tool use	❌ Won't Run	35.2GB	Q2_K	🟡 Acceptable	—
Kimi K2 1T1000B Frontier coding (multi-node)	❌ Won't Run	338.8GB	Q2_K	🟡 Acceptable	—

Can't run the model you want locally?

Compare API costs for GPT-4o, Claude, Gemini, and more — pay only for what you use.

Compare AI API Costs →

Frequently Asked Questions

🖥️ Select Your GPU

🟢

RTX 4060

NVIDIA · Ada Lovelace

GB VRAM

272

GB/s

Model

Status

VRAM

Llama 3.2 3B3B

Lightweight, edge devices

✅ Runs Great

6.5GB

Llama 3.2 1B1B

Ultra-lightweight, mobile

✅ Runs Great

2.5GB

Qwen 3 1.7B1.7B

Edge / mobile

✅ Runs Great

3.9GB

Qwen 3 0.6B0.6B

Ultra-small, IoT

✅ Runs Great

1.7GB

Gemma 4 2B2B

Edge devices

✅ Runs Great

4.5GB

Gemma 3 1B1B

Tiny, embedded

✅ Runs Great

2.5GB

DeepSeek-R1 1.5B (Distill)1.5B

Ultra-light reasoning

✅ Runs Great

3.5GB

Falcon 3 3B3B

Lightweight

✅ Runs Great

6.5GB

Falcon 3 1B1B

Edge, mobile

✅ Runs Great

2.5GB

Qwen 3 4B4B

Lightweight, fast

✅ Runs Great

4.5GB

Gemma 4 4B4B

Fast, efficient

✅ Runs Great

4.5GB

Gemma 3 4B4B

Fast, lightweight

✅ Runs Great

4.5GB

Phi-4 Mini 3.8B3.8B

Ultra-compact, on-device

✅ Runs Great

4.3GB

Yi 1.5 9B9B

Fast bilingual

✅ Runs Great

6.7GB

Qwen 3.5 14B14B

Balanced quality

✅ Runs Great

6.7GB

Qwen 3 14B14B

Balanced performance

✅ Runs Great

6.7GB

DeepSeek-R1 14B (Distill)14B

Balanced reasoning

✅ Runs Great

6.7GB

Phi-4 14B14B

Coding, STEM

✅ Runs Great

6.7GB

Qwen 3.5 7B7B

Fast general use

⚡ Tight Fit

7.6GB

Mistral 7B v0.37B

Fast, efficient

⚡ Tight Fit

7.6GB

Command R7B7B

Lightweight RAG

⚡ Tight Fit

7.6GB

Llama 3.1 8B8B

Fast, consumer GPUs

⚡ Tight Fit

7.0GB

Qwen 3 8B8B

Fast general use

⚡ Tight Fit

7.0GB

DeepSeek-R1 8B (Distill)8B

Fast reasoning

⚡ Tight Fit

7.0GB

InternLM 3 8B8B

Reasoning, tool use

⚡ Tight Fit

7.0GB

Falcon 3 10B10B

Multilingual, efficient

⚡ Tight Fit

7.5GB

Gemma 4 12B12B

Balanced quality

⚡ Tight Fit

7.3GB

Gemma 3 12B12B

Balanced quality

⚡ Tight Fit

7.3GB

Mistral Nemo 12B12B

Balanced quality

⚡ Tight Fit

7.3GB

Qwen 3.5 32B32B

Coding, math

🐌 CPU Offload

10.8GB

Qwen 3 32B32B

High quality, coding

🐌 CPU Offload

10.8GB

Gemma 4 27B27B

Reasoning, code

🐌 CPU Offload

9.1GB

Gemma 3 27B27B

Code, reasoning

🐌 CPU Offload

9.1GB

DeepSeek-R1 32B (Distill)32B

Reasoning, coding

🐌 CPU Offload

10.8GB

Mistral Small 3.1 24B24B

Fast, function-calling

🐌 CPU Offload

8.1GB

Mixtral 8x7B46.7B

MoE efficiency

🐌 CPU Offload

15.8GB

Command R 35B35B

RAG, enterprise

🐌 CPU Offload

11.9GB

Yi 1.5 34B34B

Bilingual EN/CN

🐌 CPU Offload

11.5GB

Llama 4 Maverick400B

Frontier quality, multimodal

❌ Won't Run

135.5GB

Llama 4 Scout109B

Efficient long-context

❌ Won't Run

36.9GB

Llama 3.3 70B70B

Best open-source dense

❌ Won't Run

23.7GB

Llama 3.1 405B405B

Max quality (multi-GPU)

❌ Won't Run

137.2GB

Llama 3.1 70B70B

Strong all-rounder

❌ Won't Run

23.7GB

Qwen 3.5 72B72B

Top-tier reasoning

❌ Won't Run

24.4GB

Qwen 3 235B-A22B235B

Frontier MoE

❌ Won't Run

79.6GB

Qwen 3 72B72B

Top-tier multilingual

❌ Won't Run

24.4GB

DeepSeek-R1 671B671B

Frontier reasoning (multi-GPU)

❌ Won't Run

227.3GB

DeepSeek-R1 70B (Distill)70B

Strong reasoning

❌ Won't Run

23.7GB

DeepSeek-V3 671B671B

General (multi-GPU)

❌ Won't Run

227.3GB

Mistral Large 2 123B123B

Frontier, multilingual

❌ Won't Run

41.7GB

Mixtral 8x22B141B

MoE frontier

❌ Won't Run

47.8GB

Command R+ 104B104B

Enterprise, tool use

❌ Won't Run

35.2GB

Kimi K2 1T1000B

Frontier coding (multi-node)

❌ Won't Run

338.8GB