What does quantization mean for an AI model?

Quantization stores a model's weights at fewer bits to shrink it. An 8-bit copy is about half the size of 16-bit, and 4-bit halves it again, with a small quality cost. Choosing the right quantization level is usually what makes a model fit your VRAM.

Local LLM Glossary: VRAM, GGUF, KV Cache, MoE

Q: What is the KV cache in an LLM?

The KV cache is memory that stores the attention keys and values for every token in your context. It grows with context length and sits on top of the model's weights, so a longer conversation needs more memory. AI Local Check computes it from each model's real architecture.

Q: What is a Mixture-of-Experts (MoE) model?

A Mixture-of-Experts model is split into many experts, of which only a few run per token. That makes it faster than a dense model of the same size, but every expert must be held in memory, so its memory footprint is set by the total parameter count, not the active one.

Q: What is the difference between active and total parameters?

For a Mixture-of-Experts model, total parameters are everything loaded in memory, while active parameters are the smaller number actually used per token. A 30B-A3B model has 30 billion total and 3 billion active. Active parameters buy speed, not a smaller memory footprint.

By Lefi Abdelmonem

Author · AI Local Check

Running AI models locally comes with its own vocabulary. This glossary explains the terms you'll meet on this site and in tools like Ollama and LM Studio — in plain language, with links to deeper guides. All the memory concepts here are exactly what our calculator measures.

Memory & hardware

VRAM (video RAM) — the memory on your graphics card. A model held entirely in VRAM runs fast. It's the single most important number for local AI. How much VRAM you need →

System RAM — your computer's main memory. Slower than VRAM for AI, but it can hold the parts of a model that don't fit on the GPU (see offloading).

Offloading — splitting a model between VRAM and system RAM when it's too big for the GPU alone. It works, but the parts in RAM run more slowly.

Unified memory — a single memory pool shared by CPU and GPU, as on Apple Silicon and AMD Strix Halo. Large unified-memory machines (64–128 GB) can hold models far bigger than any consumer graphics card.

Model files & size

Parameters — the learned weights of a model, counted in billions (e.g. "8B" = 8 billion). More parameters generally means more capable, and more memory.

Quantization — storing a model's weights at fewer bits to shrink it. An 8-bit copy is about half the size of 16-bit; 4-bit halves it again, with a small quality cost. Picking the right level is usually what makes a model fit. Quantization explained →

GGUF — the file format used by llama.cpp, Ollama and LM Studio to run models locally. Each GGUF file is one quantization of a model, so its size on disk closely tracks the memory it needs.

Q4_K_M, Q5_K_M, Q8_0, IQ2_XXS… — quantization labels. The number is roughly the bits per weight (higher = better quality, more memory); the letters denote the method. Q4_K_M is a popular balance. Full breakdown →

imatrix (importance matrix) — a file used during quantization to preserve the most important weights. It is not a runnable model, so we exclude it from size calculations.

Running the model

Context length — how many tokens (roughly, word-pieces) the model can consider at once. A longer context holds more of your conversation or documents, but uses more memory via the KV cache.

KV cache — memory that stores the attention keys and values for every token in your context. It grows with context length and sits on top of the model's weights. We compute it from each model's real architecture.

Tokens per second (t/s) — how fast a model generates text. It depends on your hardware and settings, so on this site speed is always a labeled estimate, never an exact figure.

Architecture terms

Mixture-of-Experts (MoE) — a model split into many "experts," of which only a few run per token. That makes it faster than a dense model of the same size — but every expert must be held in memory, so its footprint is set by the total parameter count, not the active one.

Active vs total parameters — for an MoE model, "total" is everything loaded in memory; "active" is the smaller number actually used per token. A "30B-A3B" model has 30B total, 3B active. Active parameters buy speed, not a smaller memory footprint.

Dense model — a model where every parameter is used for every token (the opposite of MoE). Simpler, and its size directly reflects its parameter count.

Grouped-query attention (GQA) — an attention design with fewer key/value heads than query heads, which shrinks the KV cache versus classic multi-head attention. Common in modern models.

Hybrid attention — only some layers keep a full KV cache; the rest use linear attention with a constant-size state. This keeps long contexts cheap in memory.

Multi-head Latent Attention (MLA) — caches a single compressed latent per layer instead of keys and values per head, keeping the KV cache small even at very long context. Used by DeepSeek and Kimi models.

Verdicts on this site

Fits in VRAM — the model runs entirely on your GPU. Fast.

Runs with offload — part of the model spills to system RAM. Works, but slower.

Insufficient — it doesn't fit in VRAM and RAM combined.

Ready to try it? Check what your PC can run, or start with how to run an LLM locally.

→ Check what your exact GPU can run