Run ibm-granite/granite-4.1-3b-GGUF locally
ibm-granite/granite-4.1-3b-GGUF is a mid-size language model with 3.4 billion parameters, built on the granite architecture. It is released under the apache-2.0 license and has been downloaded 218,936 times.
To run ibm-granite/granite-4.1-3b-GGUF locally at a 4,096-token context, its quantized versions need between 2.39 GB (Q2_K, lowest quality) and 7.45 GB (BF16, highest quality) of memory, weights plus KV cache and a system margin included.
For most users the best balance is BF16, needing about 7.45 GB. That means ibm-granite/granite-4.1-3b-GGUF fits entirely in the VRAM of a 6 GB GPU or larger, running fully on the GPU.
All quantizations
| Quant. | Bits | Quality | Weights | KV | Total | Speed~ | Verdict |
|---|---|---|---|---|---|---|---|
| Q2_K | 3.22 | Low | 1.28 GB | 0.31 GB | 2.39 GB | 313.2 t/s | Fits in VRAM |
| Q3_K_S | 3.68 | Fair | 1.46 GB | 0.31 GB | 2.57 GB | 274.1 t/s | Fits in VRAM |
| Q3_K_M | 4.06 | Fair | 1.61 GB | 0.31 GB | 2.72 GB | 248.9 t/s | Fits in VRAM |
| Q3_K_L | 4.38 | Good | 1.74 GB | 0.31 GB | 2.85 GB | 230.4 t/s | Fits in VRAM |
| Q4_0 | 4.67 | Good | 1.85 GB | 0.31 GB | 2.96 GB | 216.4 t/s | Fits in VRAM |
| Q4_K_S | 4.7 | Good | 1.86 GB | 0.31 GB | 2.97 GB | 214.9 t/s | Fits in VRAM |
| Q4_K_M | 4.94 | Good | 1.96 GB | 0.31 GB | 3.07 GB | 204.6 t/s | Fits in VRAM |
| Q4_1 | 5.13 | Very good | 2.03 GB | 0.31 GB | 3.14 GB | 196.9 t/s | Fits in VRAM |
| Q5_0 | 5.59 | Very good | 2.21 GB | 0.31 GB | 3.33 GB | 180.6 t/s | Fits in VRAM |
| Q5_K_S | 5.59 | Very good | 2.21 GB | 0.31 GB | 3.33 GB | 180.6 t/s | Fits in VRAM |
| Q5_K_M | 5.73 | Very good | 2.27 GB | 0.31 GB | 3.38 GB | 176.2 t/s | Fits in VRAM |
| Q5_1 | 6.05 | Very good | 2.4 GB | 0.31 GB | 3.51 GB | 166.8 t/s | Fits in VRAM |
| Q6_K | 6.57 | Excellent | 2.6 GB | 0.31 GB | 3.72 GB | 153.6 t/s | Fits in VRAM |
| Q8_0 | 8.51 | Excellent | 3.37 GB | 0.31 GB | 4.48 GB | 118.7 t/s | Fits in VRAM |
| BF16 | 16.01 | Excellent | 6.34 GB | 0.31 GB | 7.45 GB | 63.1 t/s | Fits in VRAM |
KV cache computed from the model's exact architecture. Speed is a rough estimate bounded by memory bandwidth.
Frequently asked questions
How much VRAM do you need to run ibm-granite/granite-4.1-3b-GGUF?
You need about 4.48 GB of VRAM to run ibm-granite/granite-4.1-3b-GGUF entirely on the GPU using the Q8_0 quantization (at a 4,096-token context). Smaller quantizations lower the requirement at the cost of quality.
Can I run ibm-granite/granite-4.1-3b-GGUF on an 8 GB GPU?
Yes. With 8 GB of VRAM you can run ibm-granite/granite-4.1-3b-GGUF fully on the GPU using BF16 (about 7.45 GB).
Can I run ibm-granite/granite-4.1-3b-GGUF on a 16 GB GPU?
Yes. With 16 GB of VRAM you can run ibm-granite/granite-4.1-3b-GGUF fully on the GPU using BF16 (about 7.45 GB).
Can I run ibm-granite/granite-4.1-3b-GGUF on a 24 GB GPU?
Yes. With 24 GB of VRAM you can run ibm-granite/granite-4.1-3b-GGUF fully on the GPU using BF16 (about 7.45 GB).
What is the best quantization for ibm-granite/granite-4.1-3b-GGUF?
If memory allows, higher bits-per-weight means better quality. A common sweet spot is a Q4_K_M or Q5_K_M quantization, which keeps most of the quality while roughly halving the memory versus 8-bit. Pick the highest quantization that still fits in your VRAM.