Best Local AI Models for NVIDIA RTX 5090 (32 GB VRAM)
The NVIDIA RTX 5090 has 32 GB of VRAM. Here are the popular AI models it can run locally (4,096-token context, ~32.0 GB system RAM assumed), ranked by popularity.
See also: Best GPU for running local LLMs.
The NVIDIA RTX 5090 is built on NVIDIA's Blackwell architecture featuring GDDR7 512-bit delivering 1792 GB/s of raw memory bandwidth. The ultimate consumer local AI powerhouse: 32 GB of ultra-fast GDDR7 runs 70B models comfortably in 4-bit quantizations.
Equipped with 32 GB of dedicated VRAM, the NVIDIA RTX 5090 can run 36 popular open-source models completely in GPU memory without offloading. This includes full-speed execution for weights like Qwen3-8B-GGUF, Qwen3-4B-GGUF, Qwen2.5-7B-Instruct-GGUF. For a comprehensive breakdown of compatible model weights, see our guide to the best LLMs for 32 GB VRAM.
With a memory bandwidth of 1792 GB/s, this card can generate tokens at an estimated peak rate of ~298.7 tokens/second on an 8B parameter model (Q4_K_M). Its rated power draw is 575W TDP, so ensure your system's power supply and case ventilation are adequate for sustained local inferencing.
36 fit fully in VRAM · 2 run with offload
| Model | Size | Quant. | Quality | Memory | Speed~ | Verdict |
|---|---|---|---|---|---|---|
| Qwen/Qwen3-8B-GGUF | 8.19B | Q8_0 | Excellent |
9.47 GB
|
220.9 t/s | Fits in VRAM |
| Qwen/Qwen3-4B-GGUF | 4.02B | Q8_0 | Excellent |
5.35 GB
|
449.5 t/s | Fits in VRAM |
| Qwen/Qwen2.5-7B-Instruct-GGUF | 7.62B | GGUF | Excellent |
15.21 GB
|
126.3 t/s | Fits in VRAM |
| unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF | 8.0B | BF16 | Excellent |
14.87 GB
|
127.8 t/s | Fits in VRAM |
| unsloth/gpt-oss-20b-GGUF | 20.91B | F16 | Very good |
13.74 GB
|
139.5 t/s | Fits in VRAM |
| bartowski/Qwen2.5-32B-Instruct-GGUF | 32.76B | Q6_K_L | Excellent |
27.19 GB
|
70.6 t/s | Fits in VRAM |
| Qwen/Qwen2.5-1.5B-Instruct-GGUF | 1.54B | GGUF | Excellent |
4.23 GB
|
540.4 t/s | Fits in VRAM |
| unsloth/LFM2.5-24B-A2B-GGUF | 23.97B | Q8_0 | Excellent |
24.62 GB
|
75.5 t/s | Fits in VRAM |
| unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF | 30.53B | Q8_0 | Excellent |
31.43 GB
|
59.2 t/s | Fits in VRAM |
| Qwen/Qwen2.5-0.5B-Instruct-GGUF | 0.49B | GGUF | Excellent |
2.03 GB
|
1519.4 t/s | Fits in VRAM |
| unsloth/Ornith-1.0-9B-GGUF | — | BF16 | Excellent |
17.61 GB
|
107.4 t/s | Fits in VRAM |
| Qwen/Qwen3-14B-GGUF | 14.77B | Q8_0 | Excellent |
16.05 GB
|
122.6 t/s | Fits in VRAM |
| Qwen/Qwen2.5-Coder-7B-Instruct-GGUF | 7.62B | GGUF | Excellent |
15.21 GB
|
126.3 t/s | Fits in VRAM |
| ggml-org/gpt-oss-120b-GGUF | 116.83B | BF16 | Very low |
2.42 GB
|
1212.6 t/s | Fits in VRAM |
| LiquidAI/LFM3-1.7B-GGUF | 1.73B | BF16 | Excellent |
4.08 GB
|
555.0 t/s | Fits in VRAM |
| unsloth/flint-GGUF | 39.79B | Q5_K_XL | Very good |
28.77 GB
|
64.3 t/s | Fits in VRAM |
| janhq/Jan-v3.5-4B-gguf | 4.41B | GGUF | Excellent |
9.59 GB
|
217.9 t/s | Fits in VRAM |
| Qwen/Qwen2.5-3B-Instruct-GGUF | 3.09B | GGUF | Excellent |
7.27 GB
|
282.9 t/s | Fits in VRAM |
| unsloth/GLM-4.7-Flash-GGUF | 31.22B | Q8_0 | Excellent |
30.66 GB
|
60.4 t/s | Fits in VRAM |
| bartowski/Meta-Llama-3.1-8B-Instruct-GGUF | 8.03B | F32 | Excellent |
31.2 GB
|
59.9 t/s | Fits in VRAM |
| unsloth/Llama-3.2-3B-Instruct-GGUF | 3.21B | F16 | Excellent |
7.23 GB
|
299.1 t/s | Fits in VRAM |
| LiquidAI/LFM2.5-1.2B-Instruct-GGUF | 1.17B | BF16 | Excellent |
3.03 GB
|
821.1 t/s | Fits in VRAM |
| lmstudio-community/gemma-3-1b-it-GGUF | 1.0B | Q8_0 | Excellent |
1.96 GB
|
1799.4 t/s | Fits in VRAM |
| bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF | 34.66B | Q6_K_L | Excellent |
29.1 GB
|
63.5 t/s | Fits in VRAM |
| ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF | 31.58B | Q4_0 | Good |
18.42 GB
|
101.8 t/s | Fits in VRAM |
| Qwen/Qwen3-0.6B-GGUF | 0.75B | Q8_0 | Excellent |
1.83 GB
|
3009.1 t/s | Fits in VRAM |
| bartowski/Qwen_Qwen3-0.6B-GGUF | 0.75B | BF16 | Excellent |
2.64 GB
|
1274.8 t/s | Fits in VRAM |
| bartowski/Llama-3.3-70B-Instruct-GGUF | 70.55B | IQ3_M | Fair |
31.96 GB
|
60.2 t/s | Fits in VRAM |
| unsloth/Qwen3-1.7B-GGUF | 1.72B | BF16 | Excellent |
4.45 GB
|
558.2 t/s | Fits in VRAM |
| LiquidAI/LFM2.5-350M-GGUF | 0.35B | BF16 | Excellent |
1.51 GB
|
2704.4 t/s | Fits in VRAM |
| Qwen/Qwen2.5-Coder-14B-Instruct-GGUF | 14.77B | GGUF | Excellent |
29.07 GB
|
65.1 t/s | Fits in VRAM |
| unsloth/Qwen3-Coder-Next-GGUF | 79.67B | IQ3_S | Low |
28.83 GB
|
64.8 t/s | Fits in VRAM |
| unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF | 8.19B | BF16 | Excellent |
16.63 GB
|
117.4 t/s | Fits in VRAM |
| bartowski/Nanbeige_Nanbeige4.3-33B-GGUF | 33.12B | Q6_K_L | Excellent |
29.93 GB
|
63.3 t/s | Fits in VRAM |
| Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF | 1.54B | Q8_0 | Excellent |
2.67 GB
|
1015.6 t/s | Fits in VRAM |
| bartowski/Qwen2.5-14B-Instruct-GGUF | 14.77B | F16 | Excellent |
29.07 GB
|
65.1 t/s | Fits in VRAM |
| unsloth/Ornith-1.5-122B-GGUF | 125.09B | Q3_K_XL | Fair |
56.55 GB
|
0.9 t/s | Offload |
| google/gemma-7b | 8.54B | GGUF | Excellent |
33.11 GB
|
1.6 t/s | Offload |
"Fits in VRAM" = fast, fully on GPU. "Offload" = part on system RAM, slower. Speed is a rough estimate from the NVIDIA RTX 5090's 1792 GB/s memory bandwidth (offloaded models are limited by system RAM).
Frequently asked questions
What is the VRAM and memory bandwidth of the NVIDIA RTX 5090?
The NVIDIA RTX 5090 features 32 GB of VRAM and a memory bandwidth of 1792 GB/s (GDDR7 512-bit). In local language model inference, VRAM determines which model sizes fit on the card, while memory bandwidth dictates how many tokens per second the GPU generates.
What is the best local AI model to run on a NVIDIA RTX 5090?
The best overall model for the NVIDIA RTX 5090 is Qwen/Qwen3-8B-GGUF using the recommended Q8_0 quantization (9.47 GB total memory). With 32 GB of VRAM, this GPU typically runs a 30–32B model at high quality, or a 70B at low bit rates at full GPU speed. Check our guide to the best LLMs for 32 GB VRAM for details.
Can the NVIDIA RTX 5090 run 70B parameter models like Llama 3.3 70B?
Yes. With 32 GB of VRAM, the NVIDIA RTX 5090 can run 70B parameter models quantized to 3-bit or 4-bit (such as IQ3_M or Q4_K_M) with moderate context windows (4k tokens). For larger context windows (16k+), pairing with a second GPU or slight CPU offloading may be required.
What power supply (PSU) do you need for the NVIDIA RTX 5090 when running local AI?
The NVIDIA RTX 5090 has a rated TDP of 575W. While local inference typically draws less power than full 3D rasterization or gaming, continuous token generation can sustain high loads. A quality power supply of at least 875W is strongly recommended.
NVIDIA RTX 5090 Head-to-Head Comparisons
Compare specs, memory bandwidth, and AI model capability against other graphics cards.
RTX 5090 32GB vs RTX 4090 24GB
32 GB GDDR7 (1,792 GB/s) vs 24 GB: what new models become possible.
RTX 5090 32GB vs RTX 3090 24GB
The new 32 GB halo card vs the classic 24 GB budget legend.
NVIDIA RTX 5090 vs NVIDIA RTX 4090
Side-by-side local AI performance and supported model comparison.