What is Quantization?
Quantization converts a model’s weights from a high-precision numeric format (like 32-bit or 16-bit floating point) into a lower-precision one (like 8-bit or 4-bit integers). Fewer bits per parameter means a smaller model footprint, faster inference, and dramatically lower VRAM requirements. The conventional wisdom was that quantization always trades some accuracy for efficiency — but that assumption is starting to break down.
Hugging Face’s recent “Quantization-Aware Healing” technique repairs key weights during the quantization process itself, letting a compressed 4-bit model actually outperform its full-precision original on certain tasks — a signal that quantization isn’t just a compromise anymore, it can be a genuine improvement. In practice, this is what makes it possible to squeeze a 70B-class model onto a single consumer GPU, and it’s the core reason edge devices (phones, Jetson boards, Raspberry Pi) can run LLMs at all. For self-hosted teams, picking the right quantization scheme (GGUF, GPTQ, AWQ) is often a bigger win than buying more expensive hardware.