What is the difference between GGUF, GPTQ and AWQ?

GGUF is the quantization format for llama.cpp, Ollama and LM Studio, and it runs on CPU, GPU or a mix of both. GPTQ is an older GPU-only 4-bit format run through libraries like AutoGPTQ. AWQ is a GPU-focused 4-bit format that protects the most important weights and is often used for serving with vLLM. For running models locally, GGUF is the most flexible choice.

Which quantization format should I use to run a model locally?

GGUF, in almost all cases. It is the only common format that runs on CPU as well as GPU, it powers the most popular local tools such as Ollama and LM Studio, and it supports the widest range of quantization levels with flexible CPU/GPU offloading. GPTQ and AWQ make more sense when serving a model on a GPU through a framework like vLLM.

Does GGUF, GPTQ or AWQ use less memory?

At the same bit-rate they are similar; a 4-bit model is roughly a 4-bit model regardless of format. The bigger factor is the quantization level you choose, such as 2-bit versus 4-bit versus 8-bit, not the format label. This site's memory figures are computed from real GGUF files, which is what local tools load.

Is AWQ better than GPTQ?

AWQ tends to preserve quality better than GPTQ at 4-bit because it is activation-aware, protecting the weights that matter most, and it is commonly used for GPU serving with frameworks like vLLM. Both are GPU-only, so for local CPU or mixed CPU/GPU use, GGUF is the more practical choice.

GGUF vs GPTQ vs AWQ: Which Format to Use

By Lefi Abdelmonem

Author · AI Local Check

When you download a quantized model, you'll see different formats: GGUF, GPTQ, AWQ, and a few others. They all shrink a model so it needs less memory, but they're built for different runtimes and hardware. Picking the right one mostly comes down to what you'll run the model with. Here's the practical difference.

The short answer

GGUF — the format for llama.cpp, Ollama and LM Studio. Runs on CPU, GPU, or a mix of both. The most flexible choice, and what this site's numbers are based on. If you're running locally on a desktop or laptop, you almost certainly want GGUF.
GPTQ — an older GPU-only format, typically 4-bit, run through libraries like AutoGPTQ or in text-generation-webui. Fast on GPU, but being replaced by newer formats.
AWQ — a GPU-focused 4-bit format that protects the most important weights ("activation-aware"), often used for serving with vLLM. Strong quality at 4-bit on GPU.

Side by side

Format	Runs with	Hardware	Best for
GGUF	llama.cpp, Ollama, LM Studio	CPU, GPU, or both	Local desktop/laptop use
GPTQ	AutoGPTQ, text-gen-webui	GPU only	Older GPU setups
AWQ	vLLM, others	GPU only	GPU serving at 4-bit
EXL2	ExLlamaV2	GPU only	Fast GPU inference, flexible bit-rates
bitsandbytes	Hugging Face Transformers	GPU	On-the-fly quantization in Python

Which should you use?

For running a model locally — the reason most people are here — GGUF is the default. It's the only format of the group that runs happily on CPU as well as GPU, it powers the most popular local tools (Ollama, LM Studio), and it offers the widest range of quantization levels, from tiny 2-bit builds to near-lossless 8-bit. Its flexibility with offloading — putting some layers on the GPU and the rest in system RAM — is exactly what lets you run models bigger than your VRAM.

GPTQ and AWQ make more sense when you're serving a model on a GPU through a framework like vLLM, where they can be very fast. AWQ in particular tends to hold quality well at 4-bit. But they don't run on CPU and offer less flexibility for mixed CPU/GPU setups.

Does the format change how much memory a model needs?

At the same bit-rate, all these formats land in a similar ballpark — a 4-bit model is roughly a 4-bit model. The bigger lever is the quantization level you choose (2-bit vs 4-bit vs 8-bit), not the format label. Because this site is built for local use, its memory figures are computed from real GGUF files, which is what Ollama and LM Studio actually load.

The bottom line

Running locally on your own machine? Use GGUF — it's the most flexible, runs anywhere, and powers the popular local tools. Consider GPTQ or AWQ only if you're serving on a GPU through a framework that prefers them. To understand the quantization levels within GGUF (Q4_K_M, Q5, Q8, the IQ quants), read GGUF quantization explained, then check which quantization fits your GPU.

→ Check what your exact GPU can run