Can you run NVIDIA Nemotron 3.5 Lightning 30B locally?

Yes, on a 24 GB GPU or larger. Measured from the real GGUF files, it needs about 20.6 GB at Q3_K_XL or 24.4 GB at Q4_K_M, so an RTX 3090, 4090 or 5090 runs it. It does not fit a 16 GB card even at 2-bit, because its Mamba-hybrid design keeps a ~19 GB memory floor. A 32 GB card handles Q5_K_M at higher quality.

Why does the 30B Nemotron need 24 GB instead of 16 GB?

Nemotron 3.5 Lightning is a Nemotron-H hybrid that mixes Mamba (state-space) layers with attention. Those Mamba components and the embeddings do not compress below a certain precision the way attention weights do, so its 1-bit and 2-bit builds all stay around 18-19 GB. The usual trick of dropping to a lower quantization to fit a smaller card does not work, leaving a practical floor near 19 GB.

How much VRAM does Nemotron 3.5 Lightning need?

Measured from the real GGUF files at a 4,096-token context: about 18.9 GB at IQ2_M, 20.6 GB at Q3_K_XL, 24.4 GB at Q4_K_M, 29.0 GB at Q5_K_M and 33.4 GB at Q8_0, including KV cache and system margin. The minimum is about 19 GB, so it needs a 24 GB card to run entirely in VRAM.

Is Nemotron 3.5 Lightning fast?

Yes. It is a Mixture-of-Experts model with 128 experts of which only about 6 run per token, roughly 3B active parameters, so it runs much faster than a dense 30B. Because only 6 of its 52 layers use attention, its KV cache stays tiny, under 0.05 GB at a 4K context and only about 0.75 GB at 131K tokens, and its native context reaches 1,048,576 tokens.

Can You Run NVIDIA Nemotron 3.5 Lightning Locally?

By Lefi Abdelmonem · LinkedIn ↗

Author · AI Local Check · Published August 19, 2026

NVIDIA's Nemotron 3.5 Lightning 30B-A3B is one of the most-downloaded new models of August 2026 — the GGUF conversions from unsloth, ggml.org, bartowski and LM Studio have pulled over a quarter of a million downloads combined in days. It's a fast Mixture-of-Experts model built for local use. But there's a catch worth knowing before you download it, and it's the kind of thing only real measurement reveals. Every number here comes from the actual GGUF files.

The catch: this "30B" wants a 24 GB card, not 16 GB

You'd expect a 30B model to fit a 16 GB GPU at a low quantization — most do. Nemotron 3.5 Lightning doesn't. Measured from the real files, even its smallest builds need about 19 GB:

Quantization	Memory to run	Fits your GPU?
IQ2_M (~2-bit)	≈ 18.9 GB	❌ 16 GB · ✅ 24 GB
Q3_K_XL	≈ 20.6 GB	✅ 24 GB
Q4_K_S	≈ 23.6 GB	✅ 24 GB
Q4_K_M	≈ 24.4 GB	✅ 24 GB (tight)
Q5_K_M	≈ 29.0 GB	✅ 32 GB
Q8_0	≈ 33.4 GB	32 GB + offload

Figures include weights, KV cache and system margin at a 4,096-token context.

Why doesn't it shrink like other 30B models?

Nemotron 3.5 Lightning isn't a standard transformer — it's a Nemotron-H hybrid, mixing Mamba (state-space) layers with attention layers. Those Mamba components, plus the embeddings, don't compress below a certain precision the way attention weights do. The result is visible in the file sizes: the 1-bit and 2-bit builds barely differ (all around 18–19 GB), because a large part of the model simply stays at higher precision no matter how aggressive the quantization.

So the usual "drop to 2-bit to fit a smaller card" trick doesn't help here. The practical floor is ~19 GB, which is why a 24 GB card — RTX 3090, 4090 or 5090 — is the real entry point.

So, can you run it?

16 GB or less: no, not even at 2-bit — the ~19 GB floor exceeds your VRAM. It would run only by offloading to system RAM, slowly.
24 GB (RTX 3090/4090/5090): yes — Q3_K_XL (~20.8 GB) or Q4_K_M (~24.5 GB, tight) fit on the card.
32 GB: comfortable, up to Q5_K_M (~29 GB) for higher quality.

The upside: it's fast, and long context is cheap

Once it fits, Nemotron 3.5 Lightning earns its name. As a Mixture-of-Experts model (128 experts, only ~6 active per token, roughly 3B active parameters) it runs far faster than a dense 30B. And because only 6 of its 52 layers use attention (the Mamba layers carry a constant-size state), its KV cache is tiny — under 0.05 GB at 4K tokens, and only about 0.75 GB even at 131K — so long context adds almost nothing on top of the weights. Its native context reaches 1,048,576 tokens (1M).

MoE reminder: only ~6 of 128 experts run per token, which is what makes it fast — but all 128 experts must be held in memory. The ~19–25 GB footprint already reflects the full model, not the active slice.

The specs

• Maker: NVIDIA · Family: Nemotron-H (hybrid Mamba-Transformer)
• ~31.6B total parameters, MoE (128 experts, ~6 active ≈ 3B active)
• Licence: NVIDIA open model licence
• Native context: 1,048,576 tokens (1M)

Check it against your GPU

See live figures for every quantization on the Nemotron 3.5 Lightning model page, or enter your own hardware to confirm the fit before you download 19+ GB. If a 24 GB card is out of reach, see the best models for 16 GB instead — plenty of capable 14–24B models that do fit.

The bottom line

Nemotron 3.5 Lightning 30B is a genuinely fast, capable MoE from NVIDIA with a 1M-token context — but its Mamba-hybrid design gives it a hard ~19 GB memory floor, so it needs a 24 GB GPU, not the 16 GB its "30B" label might suggest. If you have the VRAM, it's an excellent local model; if you don't, now you know before the download. Check exactly what your PC can run.

→ Check what your exact GPU can run