Quantization: 16-bit, 8-bit and 4-bit weights, measured

Quantization stores each weight with fewer bits by rounding it onto a small grid of allowed values: 256 with 8 bits, only 16 with 4. Weights are rounded in small groups, each with its own scale, to keep the grid fine where it matters. The question is how much the rounding hurts the model.

In Daveth Pro (AI Engineering)

What this page answers

What is LLM quantization?

Storing each weight with fewer bits by rounding it onto a small grid of allowed values, in groups that each have their own scale. Read it in “Rounding the numbers”

How much smaller is a 4-bit model?

Rounding a model's 16-bit weights to 4 bits makes its file about a quarter of the size. Read it in “Rounding the numbers”

Rounding the numbers

A weight like 0.0371946 doesn't need seven digits to do its job; 0.037 is nearly as good. Quantization stores each weight with fewer bits by rounding it onto a small grid of allowed values. With 8 bits there are 256 allowed values; with 4 bits, only 16. To keep the grid fine where it matters, weights are rounded in small groups (here, 64 at a time), each group with its own scale. The question is how much the rounding errors, added up across billions of weights, hurt the model.

Predict first. Rounding a 1.5-billion-parameter model's weights to 4 bits makes its file about a quarter of the size. What happens to how well it predicts text?

  • almost no change (under 5% worse)
  • noticeably worse (5–25%)
  • much worse (over 25%)

Answer. Its perplexity went from 4.80 to 5.97, 24.2% worse, for a file 3.8 times smaller. And 8-bit, surprisingly, did far worse: perplexity 67.14. The table below explains why.

Three precisions, measured

For each model: the published 16-bit version; PyTorch's dynamic 8-bit quantization, which really stores the big matrices as 8-bit integers; and 4-bit weights, simulated by rounding every weight onto a 4-bit grid in groups of 64 and computing with the result. (The simulation measures quality exactly; it doesn't measure 4-bit speed, which needs special software such as llama.cpp.) Quality is perplexity on 40 children's stories the models weren't tuned on: lower is better, and the gap from the 16-bit version is the cost of rounding.

ModelPrecisionWeight memoryPerplexity (lower is better)ChangeSpeed, tokens/s
Qwen2.5-Coder-0.5B-Instructbfloat16 (16-bit)0.99 GB9.21–4.4
dynamic int8 (8-bit)0.49 GB24.19+162.6%4.4
4-bit weights, simulated0.26 GB11.07+20.1%not measured
Qwen2.5-1.5B-Instructbfloat16 (16-bit)3.09 GB4.80–2.1
dynamic int8 (8-bit)1.54 GB67.14+1297.4%1.2
4-bit weights, simulated0.82 GB5.97+24.2%not measured

The surprise is 8-bit. PyTorch's dynamic 8-bit quantization raised perplexity by 163% for Qwen2.5-Coder-0.5B and 1297% for Qwen2.5-1.5B, far more than 4-bit's 20.1% and 24.2%. Fewer bits did less damage than more bits, so the bit count can't be the explanation. The method is. Dynamic quantization rounds the activations to 8 bits as well as the weights, with one scale for a whole tensor, and models like these carry a few activation values far larger than the rest: one scale stretched to fit them leaves almost no resolution for everything else. The 4-bit version here rounds only the weights, with a separate scale for every 64 of them. That's most likely why practical 8-bit methods such as LLM.int8 keep outlier features in 16-bit, and why GGUF and GPTQ quantize weights in small groups. How you quantize matters as much as how many bits.