Quantization is the single technique that moved language models from data centres onto phones. Everything else — unified memory, efficient frameworks, better small models — helped. Quantization is what made it arithmetically possible.
The idea is almost embarrassingly simple: store each number in the model using fewer bits. The consequences are not simple at all, and they’re worth understanding, because quantization is why the model you download is 1.6GB instead of 4GB, and why it runs at conversational speed instead of a word every two seconds.
This article sits under How LLMs Actually Run on Your iPhone, which covers the full inference path.
What a weight actually is
A language model is a very large collection of numbers. Each one — a weight — is a value learned during training that determines how strongly one part of the network influences another. “9B parameters” means nine billion such numbers.
During training, these are usually stored as 16-bit floating point values: 16 binary digits each, giving fine gradations across a wide range. That precision genuinely matters while training, because updates are tiny and rounding them away would stall learning.
Nine billion weights at 16 bits each is 18 gigabytes. That is the number that has to come down.
The core insight: precision that training needed, inference doesn’t
Here is the observation the whole field rests on. A trained model’s weights cluster tightly around zero, and the network’s output is remarkably tolerant of small perturbations in individual weights. Nudge one weight by a fraction of a percent and the prediction is unchanged. The information is spread redundantly across billions of parameters.
So instead of storing every weight at full precision, you store an approximation.
Take a group of weights and find their range — say −0.4 to +0.4. With 4 bits you have 16 possible values, so slice that range into 16 steps and store each weight as whichever step is nearest, plus one shared scaling factor for the group. To use them, multiply back out.
You have lost information. A weight of 0.137 might come back as 0.133. Multiplied across nine billion weights, though, the errors are small, roughly unbiased, and largely cancel in aggregate.
Modern schemes are considerably more sophisticated than that sketch — the widely-cited GPTQ and AWQ papers use calibration data to work out which weights are most sensitive and protect them specifically. The principle stays the same.
What the numbers look like in practice
Applied to a 9B model:
| Precision | Bits per weight | Approx. size | Typical quality |
|---|---|---|---|
| FP16 | 16 | ~18GB | Reference |
| Q8 | 8 | ~9GB | Indistinguishable |
| Q4 | ~4.5 | ~5.6GB | Very close; small losses on precision work |
| Q3 | ~3.5 | ~4.2GB | Noticeable degradation |
| Q2 | ~2.5 | ~3GB | Frequently unusable |
Two things stand out.
4-bit is the sweet spot, and it isn’t close. The drop from 16 to 4 bits costs a few percent on benchmarks. The drop from 4 to 2 costs far more than the file size saves. Below 4 bits, models start producing subtly broken output — repetition, dropped instructions, confident nonsense.
“4-bit” is never exactly 4 bits. Group scaling factors have to be stored too. A model advertised as 4-bit typically averages 4.3–4.7 bits per weight in practice. This is why quantized model sizes look slightly odd — Qwen 3.5 2B lands at 1.6GB rather than a tidy 1.0GB.
The counter-intuitive part: it makes models faster
Quantization is usually explained as a size optimization. On a phone, the speed effect is arguably more important.
Generating a single token requires reading essentially every weight in the model. Not a clever subset — all of them, for every token. Modern chips perform arithmetic far faster than they can move numbers out of memory, so inference is bound by memory bandwidth, not by computation.
Fewer bits per weight means fewer bytes moved per token. A 4-bit model reads roughly a quarter of the bytes its 16-bit original would, and generation speed scales with it — typically around 3× faster in practice, once dequantization overhead is accounted for.
Quantization is that rare optimization with no real trade-off axis against speed: the compressed model is both smaller and faster, and only marginally less accurate.
Where the loss actually shows up
Quality loss from 4-bit quantization is not evenly distributed. It concentrates in predictable places:
- Long arithmetic chains. Multi-step calculation is unforgiving; a small error early propagates.
- Exact reproduction. Verbatim quotes, precise API signatures, exact syntax.
- Rare knowledge. Facts encoded in few weights are more fragile than facts encoded redundantly.
- Very long contexts. Small errors accumulate across thousands of tokens.
Where it barely shows up: ordinary conversation, summarization, rewriting, translation, brainstorming, explanation, and general question answering. Which is most of what people actually do.
There is also a well-established practical rule worth internalizing: a quantized larger model beats an unquantized smaller one. Given a fixed memory budget, spend it on parameters rather than precision. A 4-bit 4B model comfortably outperforms a 16-bit 1B model that occupies similar memory.
How this shapes Cloaked’s model library
Every model in Cloaked ships quantized, which is why the catalogue spans 317MB to 5.6GB rather than 1.2GB to 18GB.
That range is the whole point. Qwen 3 0.6B at 317MB runs on hardware that could not load an unquantized 0.6B model. Qwen 3.5 9B at 5.6GB runs on an iPhone 15 Pro, where 18GB would be impossible on any phone made to date.
For choosing between them, Best Local LLM Models for iPhone compares the catalogue by size and task, and Small Language Models makes the case that the smaller end is more capable than its parameter count suggests.
The mental shift is this: quantization moved the question from “can a phone run this at all” to “which trade-off do you want.” That is a much better question to be arguing about.
Download Cloaked on the App Store and try a quantized model on your own device — every model in the library runs fully offline, with no account required.
Frequently asked questions
Does quantization make a model dumber?
Slightly, and less than most people assume. Modern 4-bit quantization typically costs 1–3% on standard benchmarks, which is rarely noticeable in conversation. The loss appears first on precision-sensitive work — long arithmetic chains, exact code, rare factual recall — and grows sharply below 4 bits.
What does Q4 mean?
Q4 means roughly 4 bits are used to store each weight, versus 16 bits in an unquantized model. The 'roughly' matters: real 4-bit schemes store extra scaling values per group of weights, so the true average is nearer 4.5 bits per parameter.
Is a quantized 9B model better than a full-precision 2B model?
Almost always yes. Parameter count dominates precision across this range. A 4-bit 9B model beats a 16-bit 2B model on nearly every benchmark while using less memory than the unquantized 2B would at full precision.
Does quantization make models faster?
Yes, and this surprises people. On-device inference is limited by memory bandwidth, not arithmetic. Fewer bits per weight means fewer bytes read per token, so a 4-bit model typically generates roughly three times faster than the same model at 16-bit.