A GGUF quant name tells you roughly how many bits each weight is stored in, and that number — bits per weight — is what sets your file size and VRAM. Q8_0 is about 8 bits per weight, Q5_K_M about 5.5, and Q4_K_M about 4.83, not the flat 4.0 the name suggests. The extra comes from K-quant block scales and higher-precision tensors mixed in. Multiply parameters by bits per weight, divide by 8, and you have the file size. The quantization size calculator does it for you.
What “quantization” means here
A full-precision model stores each weight as a 16-bit number (FP16 or BF16). That is accurate but heavy: a 7-billion-parameter model at 16 bits is about 14 GB. Quantization shrinks each weight to fewer bits so the model fits on smaller hardware. Q8 uses roughly 8 bits, Q4 roughly 4. Fewer bits means a smaller file and less VRAM, at some cost to quality.
GGUF is the file format used by llama.cpp and its ecosystem. The suffixes — Q4_K_M, Q5_K_M, Q8_0 — describe the quantization scheme, not just the bit count.
Why Q4_K_M is 4.83 bpw, not 4.0
The nominal number in the name is the base bit width. The real average is higher because of two overheads:
- Block scales. K-quants group weights into blocks and store a scale (and sometimes a min) per block so values can be reconstructed. Those scales are extra bits spread across every weight, so a “4-bit” scheme averages a bit more than 4.
- Mixed precision (the letter). The trailing
_S,_M,_Lmeans small, medium, large. TheMandLvariants keep certain sensitive tensors — attention and feed-forward layers that hurt quality most when crushed — at a higher bit width. That raises the average.
So Q4_K_M lands around 4.8 to 4.85 bits per weight, Q4_K_S a little lower near 4.5, and Q5_K_M around 5.5. Treat these as close approximations; exact figures vary slightly by model architecture and llama.cpp version.
A bits-per-weight and size table
Approximate bits per weight and resulting file size for a 7B-parameter model (size = params × bpw ÷ 8). Numbers are indicative and shift by model and tooling version.
| Quant | Approx. bits/weight | 7B file size | Quality notes |
|---|---|---|---|
| Q2_K | ~3.35 | ~2.9 GB | Heavy loss, emergency fit only |
| Q3_K_M | ~3.9 | ~3.4 GB | Noticeable loss |
| Q4_K_S | ~4.5 | ~3.9 GB | Good size, small quality dip |
| Q4_K_M | ~4.83 | ~4.2 GB | Common sweet spot |
| Q5_K_M | ~5.5 | ~4.8 GB | Better quality if it fits |
| Q6_K | ~6.6 | ~5.8 GB | Very close to full quality |
| Q8_0 | ~8.5 | ~7.4 GB | Near-lossless, largest |
The pattern: each step up in bits adds size and quality. Q4_K_M sits where most people stop, because the quality gain above it is small relative to the extra gigabytes.
Computing file size and VRAM for any model
The formula is simple:
File size (bytes) = parameters × bits per weight ÷ 8
For a 13B model at Q4_K_M: 13,000,000,000 × 4.83 ÷ 8 ≈ 7.85 GB on disk.
VRAM is the file size plus room for the context (KV cache) and a little overhead. A rough rule is file size plus 10 to 20 percent for a modest context, more if you run a large context window. That same 13B Q4_K_M model wants roughly 9 to 10 GB of VRAM in practice, which is why it fits a 12 GB card but not an 8 GB one.
Picking a quant that fits your card
Work backward from your VRAM:
- List your usable VRAM. Subtract 1 to 2 GB for the OS, display, and context. An 8 GB card gives you roughly 6 to 7 GB for weights.
- Find the largest quant that fits. For that 8 GB card and a 7B model, Q5_K_M (~4.8 GB) fits comfortably; Q8_0 (~7.4 GB) is too tight once context is added.
- Prefer the
_Mmedium mix. It protects the tensors that matter most, so Q4_K_M usually beats a same-size alternative on quality. - Leave headroom for context. A long context window grows the KV cache and can push you over the edge even if the weights fit.
Once the model fits, the next question is speed — smaller quants can run faster because they move less memory per token. See tokens per second explained for how quantization and hardware set generation speed, and if you are weighing local hardware against a hosted API, self-hosting vs API — when is it cheaper? covers the cost side.
To skip the arithmetic, put your parameter count and target quant into the quantization size calculator and it returns the file size and estimated VRAM directly.
