NVIDIA Nemotron 3.5 Bolt E4B GGUF Size & VRAM Requirements

NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF is a large language model focused on general purpose & instruction. With approximately 8.0 billion parameters, it balances local hardware requirements with reasoning and generation fidelity. It is built on the gemma4 architecture with a native context window of up to 131,072 tokens. The model is primarily recommended for general knowledge, document summarization, email drafting, language translation, and multi-turn conversational chat. Released under the other license, it can be executed fully offline without sending data to external APIs.

Under the hood it uses 42 transformer layers, a hidden size of 2,560, 8 attention heads. It uses a hybrid attention design: only 7 of its 42 layers keep a full KV cache, while the rest use linear attention with a constant-size state. Long contexts therefore cost far less memory than on a conventional model of this size.

To run unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF locally at a 4,096-token context, its quantized versions need between 1.23 GB (Q8_0, lowest quality) and 14.87 GB (BF16, highest quality) of memory, weights plus KV cache and a system margin included. Context length drives that memory directly: at 4,096 tokens the KV cache for NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF is about 0.05 GB, rising to roughly 1.75 GB at its full 131,072-token context. Shorter prompts free up memory for a higher-quality quantization.

For most users the best balance is Q6_K_XL, needing about 7.62 GB. That means unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF fits entirely in the VRAM of a 6 GB GPU or larger, running fully on the GPU.

Available GGUF quantizations for unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF include Q8_0, IQ2_M, IQ3_XXS, Q2_K_XL, Q3_K_XL, Q4_K_XL, Q5_K_XL, Q6_K_XL, Q8_K_XL, BF16. The model supports a native context length of up to 131,072 tokens; a longer context grows the KV cache and the memory needed.

→ Guide: How much VRAM do you need?