How Much VRAM Do You Need to Run an LLM?

VRAM (Video Random Access Memory) is the most critical resource when running Large Language Models locally. This guide breaks down exactly what you need.

The Simple Formula

For a model to fit in GPU memory, you need roughly:

- 4 bytes per parameter for FP32 (full precision)

- 2 bytes per parameter for FP16/BF16 (half precision)

- 1 byte per parameter for INT8 (8-bit quantization)

- 0.5 bytes per parameter for INT4 (4-bit quantization)

VRAM by Model Size

-------------------------

Practical Recommendations

- 7B models (Q4): Any modern GPU with 8GB+ — RTX 3060, A4000

- 13B models (Q4): 12-16GB VRAM — RTX 3080, A5000

- 70B models (Q4): Multi-GPU or cloud — or use a provider like Groq

- 33B models (Q4): 24GB VRAM — RTX 3090/4090

KV Cache

Don't forget the KV cache! It also consumes VRAM proportional to context length. A 4096-token context can add several GB overhead on large models.

Use the VRAM Calculator to estimate exactly what you need for your specific setup.

The Bottom Line

With 24GB VRAM, a Q4 estimate for many 7B–32B models lands inside the budget, and the 2026 27B-class generation targets that tier. For 70B models, consider cloud GPUs or providers like Groq. A memory estimate is not a fit determination and not proof the model runs your workload: runtime, context length and harness change the real result.