If you've tried to download a local model, you've hit the wall: the same model is offered as GGUF, GPTQ, and AWQ, and nobody tells you which to grab. They're all quantization formats — ways of compressing a model's weights so it fits in less memory — but they're built for different runtimes and different hardware. Pick the wrong one and it either won't load or runs badly. This is the plain-language comparison, and by the end you'll know which to click without a computer-science degree.
A model's weights are normally 16-bit numbers. Quantization stores them in fewer bits — commonly 4 — which shrinks the file by roughly three-quarters and lets a model that needed a big GPU run on modest hardware, with a small and usually acceptable quality cost. If you want the full mechanism, LLM quantization explained covers it. The three formats below are just different recipes for doing that compression, each optimized for a different way of running the model.
GGUF is the format for running models on CPUs, laptops, and mixed CPU-plus-GPU setups. It's a single self-contained file, it's what the popular desktop runners expect, and it's the most forgiving choice if you don't have a big dedicated GPU. It offers a whole ladder of quant levels so you can trade size for quality. If you're running a model on a normal computer, or you're not sure what hardware you'll end up on, GGUF is the safe default — it's the same format behind running an LLM on CPU with no GPU. Deeper background lives in what is a GGUF model.
GPTQ is a GPU-oriented format that quantizes a model using a calibration dataset to preserve accuracy at 4 bits. It's been around long enough to be widely supported by GPU inference servers, and it's a solid pick when your model lives entirely in GPU memory and you want fast generation. Its weakness is the flip side of its strength: it's built for the GPU path, so it's not the right choice if you're trying to run mostly on CPU. If you have a capable GPU and want throughput, GPTQ is a reasonable choice.
AWQ is the more recent GPU format. Its idea is that not all weights matter equally, so it protects the most important ones during quantization — which tends to preserve quality well at 4-bit while staying fast on GPU. In practice it's become a favorite for GPU serving where you want strong quality without giving up speed. Like GPTQ, it assumes a GPU; it is not aimed at CPU-only setups. If you're deploying on GPUs and quality-per-token matters, AWQ is worth testing head-to-head against GPTQ on your own workload.
No GPU, a laptop, or unsure hardware → GGUF. A real GPU and you want fast serving → GPTQ or AWQ, with AWQ often the better quality-at-speed pick and GPTQ the more universally supported one. Whatever you choose, the amount of memory it needs depends on the model size and the bit-width — check it before you download with the VRAM calculator so you know a 4-bit build actually fits your card or your RAM. And estimate how fast it'll generate with the inference speed calculator before you build anything around it.
People want a single winner, but at the same bit-width the quality differences between these formats are usually small compared to the difference between model sizes. A bigger model at 4-bit almost always beats a smaller model at 8-bit. So don't agonize over the format — get the largest model that fits your memory (the calculators above tell you what fits), pick the format that matches your hardware, and spend your energy on the prompt and the retrieval instead. To choose the underlying model in the first place, see the best local AI models of 2026.
The format is a hardware question, not a hard question. The sizing tools above are free, no signup — they tell you what fits before you download 20GB. QADIR OS runs local models directly, so once you've picked a build it plugs straight in as an on-device brain. Join QADIR OS early access.