Quantization

What is Quantization?

Quantization is a foundational model compression technique in machine learning that converts high-precision numerical representations—typically 32-bit or 16-bit floating-point numbers—into lower-precision formats like 8-bit or 4-bit integers. By simplifying the math that powers a model, quantization significantly reduces its size, memory footprint, and the computational cost required to run it, though this usually comes at the cost of a small loss in accuracy.

In the world of large language models, these models are often massive, requiring enormous amounts of memory to store and run. Quantization acts as a way to shrink these digital giants so they can fit onto more accessible hardware, such as standard GPUs or even consumer-grade devices, without losing their core intelligence.

Quantization preserves the full structure of a model while reducing the precision of its numbers—like compressing a photo without cropping it.

A Simple Analogy

Think of quantization like converting a high-resolution photograph into a smaller file size. A RAW photo, much like a model in high-precision FP32 format, contains millions of color gradations and immense detail. A JPEG, which acts like an INT8 quantized model, uses fewer colors but looks nearly identical to the human eye. A heavily compressed thumbnail, similar to an INT4 model, loses some fine detail but remains perfectly recognizable and loads much faster. Just as you would not need a 50MB RAW file to share a quick photo on social media, you often do not need full-precision math to run an efficient AI model.

How It Works

At its core, quantization maps real-valued tensors—the complex arrays of numbers that make up a model’s weights and activations—onto a discrete integer grid using a scaling factor. Once the numbers are mapped to this grid, the computer can perform operations using integer arithmetic. Integer math is fundamentally faster and more energy-efficient for hardware to process than the more complex floating-point arithmetic used during the initial training phase.

Precision Levels: INT8 vs. INT4

The degree of compression depends on the bit-width chosen:

• INT8 (8-bit integer): Uses 256 distinct values (ranging from -128 to +127). Widely supported across CPUs, GPUs, and neural processing units (NPUs), INT8 typically achieves up to a 4x reduction in memory usage compared to FP32. It offers a reliable balance between compression and accuracy and is often considered the standard precision target for production deployment.
• INT4 (4-bit integer): A more aggressive approach using only 16 distinct values (-8 to +7). INT4 achieves roughly an 8x reduction in size compared to FP32. Because it is so compressed, it requires specialized software kernels like GPTQ or AWQ to function effectively, and it generally results in a larger drop in accuracy than INT8.

Methods of Quantization

There are two primary approaches:

• Post-training quantization (PTQ): Applied after the model has finished training. PTQ requires no retraining, making it a convenient way to compress existing models. GPTQ (Frantar et al., 2022) is a leading PTQ method that can quantize a 175-billion-parameter model to 3-4 bits in roughly four GPU hours with negligible accuracy loss.
• Quantization-aware training (QAT): Simulates the effects of quantization during the training process itself, allowing the model to learn to compensate for precision loss. QAT generally produces higher accuracy than PTQ but requires access to the training pipeline.

A third approach, Activation-aware Weight Quantization (AWQ), takes a different path: it identifies the 1% of weight channels that matter most by examining activation patterns, then protects those channels during quantization. This approach won the MLSys 2024 Best Paper Award and often achieves better accuracy than GPTQ at similar bit-widths.

For fine-tuning quantized models, QLoRA (Dettmers et al., 2023) introduced a technique that freezes a 4-bit quantized base model and trains small adapter layers in higher precision. This allows fine-tuning a 65-billion-parameter model on a single 48GB GPU while preserving full 16-bit fine-tuning performance.

Why It Matters for Deployment

Quantization is essential for moving AI from massive data centers into the real world. By reducing the memory footprint, it allows developers to run sophisticated models on hardware with limited VRAM. This is critical for inference—the process of using a trained model to make predictions—where speed and energy efficiency are paramount. Without quantization, many of the powerful models available today would be too large or too slow to run outside of specialized, high-cost environments.

Key Trade-offs

The primary trade-off is between efficiency and precision. While you gain significant savings in memory and compute costs, you introduce quantization error—a small degradation in the model’s output quality. Lower bit-widths save more memory but generally lead to more accuracy loss. Some models are more sensitive to quantization than others, so the impact varies by architecture.

Quantization is distinct from other compression methods. Pruning removes weights entirely, making the model smaller but structurally different. Distillation trains a new, smaller model to mimic the behavior of a larger one. Quantization, by contrast, preserves the full structure of the original model—it just uses fewer bits to represent each number.