Your AI Is a Hoarder. Quantization Is the Intervention.

On why trillion-parameter models need a strict diet, and how we trick them into going on one.

By Amal V S, Backend & AI Systems Architect · Wed May 06 2026

On why trillion-parameter models need a strict diet, and how we trick them into going on one.

Imagine you hired a genius consultant. She has memorized every book ever written, speaks 47 languages, and can explain quantum mechanics to a five-year-old. But there's one problem: she travels with 73 suitcases. She needs two cargo planes just to show up to a meeting. And she charges $40,000 an hour just to stand by.

That's a modern Large Language Model. Brilliant. Impractical. Obscenely expensive to run.

Quantization is the art of saying: "You're getting one carry-on. Make it work."

First, what exactly does a model "weight"?

Every LLM is, at its core, a massive pile of numbers called weights — billions of floating-point values that encode everything the model has learned. GPT-style models have tens or hundreds of billions of these. Each weight, stored at full 32-bit floating point (FP32) precision, takes up 4 bytes of memory.

Do the math on a 70-billion parameter model:

Quick math you didn't ask for:

70,000,000,000 parameters × 4 bytes = 280 GB of RAM. The latest MacBook Pro has 128 GB max. So no, you cannot "just run it locally." Not at full precision, anyway

Sarcasm Mode
"Just spin up a 70B model on your laptop!" — said every AI influencer who has never actually tried it. You'd need to duct-tape three top-end GPUs together and pray to the CUDA gods.

Enter quantization: the Marie Kondo of model sizes

Quantization is the process of reducing the numerical precision of a model's weights. Instead of storing each number as a 32-bit float (which can represent values like 3.14159265358979…), you store it as a smaller format — 16-bit, 8-bit, or even 4-bit integers.

Yes, you're intentionally making the numbers less precise. And yes, the model still works. That's the beautiful, slightly unsettling punchline.

For a 70B model, going from FP32 to INT4 means shrinking from 280 GB down to ~35 GB. That's the difference between needing a data center and needing a single good GPU. Or a very recent Mac.

But wait !, Aren't we just... rounding numbers?

Essentially, yes. And this is where your instincts will betray you.

Your brain says: "If you round enough numbers, surely the answers get terrible." But here's the thing — neural networks are stupendously redundant. They have billions of parameters specifically because no single weight is critical. The information is spread across the entire network like peanut butter on a very large slice of bread.

The analogy you didn't know you needed
Imagine a song recorded at 32-bit audio. Now compress it to 16-bit. Can you hear the difference? Maybe, with $4,000 audiophile headphones in a soundproof room. Your iPhone speaker playing at a coffee shop? Absolutely not.

Models are similar. A lot of precision is just... noise tolerance. The model learned to be robust, and that robustness extends to its own weights being slightly imprecise.

Sarcasm Mode:
"But won't it lose intelligence?" Sir, the model was trained on Reddit. It has been exposed to things no amount of quantization could possibly make worse.

Two flavors: Post-Training vs. Quantization-Aware Training

There are two main approaches to quantization, and they differ in roughly the same way that a surprise diet and a planned diet differ.

Post-Training Quantization (PTQ) is the surprise diet. You take a fully trained model, slap a lower bit-width on it, and hope for the best. It's fast, easy, and works surprisingly well most of the time. It's what tools like llama.cpp and GGUF formats use to let you run models on consumer hardware

FP32
PTQ in plain English
You've spent 5 years mastering calculus, linear algebra, and fluid dynamics. On your way to the exam, someone grabs your brain and says "sorry, we're rounding all your memorized numbers to the nearest 10." You do okay. A bit shaky on edge cases. But you pass.

Quantization-Aware Training (QAT) is the planned diet. The model is trained from the start knowing it will be quantized — it builds tolerance into its weights. The results are better, but it's slower and requires access to the original training pipeline.

INT8
QAT in plain English
You spent 5 years studying calculus, but your professor told you on day one: "By the way, in the final exam, you can only use integers." So you developed integer-friendly intuitions. You're not just tolerating the constraint — you're optimized for it.

"The remarkable thing isn't that quantization loses some quality. It's that cutting 87% of the memory footprint loses barely 6% of the quality."

The sarcastic field guide: types of quantization

Not all quantization is equal. Here's what's actually out there in the wild:

GPTQ :- One of the most popular PTQ methods. It quantizes layer by layer, using a calibration dataset to minimize the error introduced by rounding. Think of it as rounding your weights, but checking your work after every step.

Sarcasm Mode:
GPTQ stands for "Generative Pre-trained Transformer Quantization," which is just a fancy way of saying "we compressed it and it mostly still works." The paper has 14 pages of math to say exactly that.

GGUF (via llama.cpp) :- The people's champion. A file format + quantization scheme that lets you run models on your CPU, your Mac, your laptop. The runtime is written in C++, has no Python dependencies, and will run on a Raspberry Pi if you're patient enough. It is democracy in model deployment.

AWQ (Activation-aware Weight Quantization) :- Smarter than GPTQ. Instead of treating all weights equally, AWQ identifies which weights matter most (by looking at activations) and protects them from heavy rounding. Lossy compression, but targeted. Like a JPEG algorithm that keeps faces sharp but blurs the sky.

bitsandbytes (4-bit NF4) :- Used heavily in QLoRA fine-tuning. Loads a model at 4-bit precision with special "NormalFloat4" values optimized for neural network weight distributions. The name "bitsandbytes" sounds like a startup that sells artisanal USB drives, but it's actually serious engineering.

The real-world stakes: why this actually matters

This isn't just an academic exercise. Quantization is the reason AI is accessible at all outside of big tech companies.

Without it: LLaMA 3 70B requires 8× A100 GPUs to run inference. That's ~$30,000 in hardware, plus $5/hour in cloud costs.

With 4-bit quantization: it runs on a single RTX 4090 (~$1,600). Or a Mac Studio (~$1,999). Or, with CPU offloading, a gaming PC with enough RAM.

Sarcasm Mode:
Silicon Valley circa 2022: "AI will remain exclusively in the hands of companies with $500M compute budgets." Also Silicon Valley, 2024: some guy in Kerala is running a 70B model on his MacBook and asking it to write his cover letters.

Quantization is the great democratizer. It's why open-source models can compete with proprietary APIs. It's why your laptop can now do things that, three years ago, required a server rack.

Where quantization breaks down (honesty time)

It's not perfect. A few places where quantization genuinely hurts:

Long-form reasoning: Tasks requiring chains of precise logic — complex math, code with tricky edge cases — can degrade noticeably at aggressive quantization levels (2-bit, 3-bit).

Low-resource languages: Models see less data for minority languages during training, so their representations are already less robust. Quantization error compounds on already-thin knowledge.

Smaller models: A 7B model at 4-bit loses a higher relative share of capacity than a 70B model at 4-bit. Size is actually a hedge against quantization loss.

Sarcasm Mode:
Running a 2-bit quantized 7B model for complex financial analysis is like hiring a consultant, firing 94% of their brain cells, and then being surprised when the spreadsheet has errors. Some corners genuinely cannot be cut.

So what should you actually use?

Here's the practical answer most blogs are too cowardly to give:

For running models locally: use GGUF Q4_K_M or Q5_K_M via llama.cpp or Ollama. It's the best quality-to-size ratio for consumer hardware. Q4_K_M of a 70B model fits in 40 GB and performs within a rounding error of FP16.

For fine-tuning on consumer GPUs: use bitsandbytes with QLoRA. Load in 4-bit, train only the adapters. You can fine-tune a 13B model on a single RTX 3090.

For production serving at scale: use AWQ or GPTQ with vLLM. Better throughput, GPU-optimized kernels, real batching support.

The actual takeaway
→ LLMs are enormous by default — hundreds of GB at full precision.
→ Quantization reduces numerical precision of weights, shrinking models 4–8×.
→ Neural networks are redundant enough that ~4% quality loss at 4-bit is the norm.
→ Post-training methods (GPTQ, AWQ, GGUF) work without retraining. QAT requires training from scratch but wins on quality.
→ This is what makes local AI possible. Without it, we'd all be renting GPUs from Jeff  Bezos forever.