This is the practical walkthrough — the do-this-then-that sequence for taking a full-precision model and producing a working quantized version you can deploy. No theory detours. If you can run a Python script and have a model you want to compress, you can follow every step here today.
We will quantize a mid-sized open model to 4-bit using AWQ as the running example, because it offers a strong quality-to-effort ratio. The same shape of process applies to GPTQ, GGUF k-quants, and INT8. Where the steps differ by method, we call it out.
Before you start, confirm you have the original model weights, a GPU with enough memory to load the full-precision model briefly, and a handful of representative text samples from your domain.
Step 1: Decide Your Target Precision
Pick the bit width before you touch any code, because it dictates everything downstream.
- INT8 if you want near-lossless quality and have hardware with strong 8-bit kernels.
- 4-bit (AWQ or GPTQ) if you need to fit a large model on limited memory and can tolerate a small quality dip.
- GGUF k-quants if your deployment target is CPU or mixed CPU/GPU via llama.cpp.
For this walkthrough we choose 4-bit AWQ. If you are unsure why these formats differ, The Complete Guide lays out the precision landscape.
Step 2: Assemble A Calibration Dataset
Post-training quantization needs a small sample of real text to measure how values are distributed. This is the most undervalued step.
- Collect 128 to 512 short samples that resemble your actual production inputs.
- Match the domain. If you serve legal queries, calibrate on legal text, not generic web data.
- Avoid duplicates and avoid one giant document; variety matters more than volume.
Poor calibration data is the leading cause of avoidable quality loss. The common mistakes guide covers how badly this can go.
Step 3: Install The Tooling
Set up a clean environment so dependency conflicts do not derail you.
- Create a fresh virtual environment with a recent Python.
- Install the quantization library — AutoAWQ for this example, or AutoGPTQ for GPTQ, or build llama.cpp for GGUF.
- Verify your GPU drivers and that the framework sees the GPU before proceeding.
Step 4: Run The Quantization Pass
Now execute the compression.
Load And Configure
Load the full-precision model and point the quantizer at your calibration set. Set the bit width to 4, choose a group size (128 is a sensible default — smaller groups mean better quality but slightly larger files), and enable any method-specific options.
Execute And Save
Run the quantization. For a seven-billion-parameter model this takes roughly ten to forty minutes on a single capable GPU. The tool walks layer by layer, computes the low-precision representation, and writes out new weights plus the scale factors needed to reconstruct values at runtime. Save the result in the format your serving stack expects.
Step 5: Verify Quality Before You Trust It
Never ship a quantized model on faith. Measure it.
- Perplexity check — Run perplexity on a held-out text sample for both the original and the quantized model. A small increase is normal; a large jump signals a problem.
- Task evaluation — Run your actual downstream tasks. If you do summarization, score summaries. If you do classification, check accuracy.
- Spot-check hard cases — Manually test edge cases, multi-step reasoning, and instruction-following, where quantization damage hides.
If quality is unacceptable, the best practices guide covers recovery options like better calibration, a higher bit width, or moving to quantization-aware training.
Step 6: Benchmark Speed And Memory
Confirm you actually got the wins you quantized for.
- Measure peak GPU memory loaded — it should drop by roughly the precision ratio.
- Measure tokens per second on your target hardware, not a dev box.
- Watch for the trap where a poorly supported kernel makes the quantized model slower than FP16 despite using less memory.
Step 7: Deploy And Monitor
Ship it, then keep watching.
- Roll out behind a flag so you can compare against the full-precision version in production.
- Track quality-sensitive metrics — escalation rates, user thumbs-down, downstream error rates.
- Keep the original weights available so you can roll back instantly if quality regresses.
For a fuller decision tool, the 2026 checklist turns these steps into a tick-box list you can reuse on every model.
Adapting The Process For Other Methods
The seven steps stay the same in shape; only the conversion details change. Knowing the differences saves you from re-learning the process for each format.
For GPTQ
GPTQ uses the same calibration-driven, layer-by-layer approach as AWQ, but compensates for accumulated error using second-order information. Set bit width and group size identically. The output integrates broadly across GPU serving stacks, which is its main advantage.
For GGUF (CPU/Edge)
Instead of a Python quantization library, you build llama.cpp and run its conversion utility, choosing a k-quant variant like Q4KM. There is no GPU requirement, though conversion is slower on CPU. This is the path when your target is a laptop, an edge device, or offline use.
For INT8
INT8 is often the gentlest conversion and can sometimes skip extensive calibration. It shines when you have hardware with strong integer kernels and want near-lossless quality with real throughput gains.
A Common Failure To Watch For
The most frequent way this process goes wrong is silent degradation that passes the perplexity check in Step 5 but fails real tasks. If your perplexity looks fine but spot-checks feel off, do not ship.
- Re-examine your calibration data first — generic data is the usual culprit.
- Try a higher bit width or smaller group size to recover quality.
- If neither works at your target precision, the honest answer may be that this model needs quantization-aware training to hit your bit width.
Catching this before deployment, not after, is the entire reason Step 5 exists.
Frequently Asked Questions
How long does quantizing a model take?
For a seven-billion-parameter model, expect ten to forty minutes on a single capable GPU for PTQ methods like AWQ or GPTQ. Larger models scale up roughly with parameter count. Quantization-aware training takes far longer because it involves actual training steps.
Do I need the full-precision model to quantize?
Yes, for post-training quantization you need to load the original weights once during the conversion. After that you can discard them from your serving environment, though it is wise to archive them for rollback.
What if I don't have domain-specific calibration data?
You can use a general text corpus and still get reasonable results, but quality on your specific tasks will usually be better with in-domain samples. Even a few hundred representative examples meaningfully improve outcomes.
Can I quantize on CPU instead of GPU?
GGUF conversion via llama.cpp can run on CPU, though slowly. GPU-based methods like AWQ and GPTQ effectively require a GPU because they load and process the full model. Match the method to the hardware you have.
How do I know which bit width is "good enough"?
Define a quality threshold from your task evaluation before quantizing — for example, no more than a two-point drop on your benchmark. Then try the most aggressive bit width that stays under that threshold. Let measured quality, not a default, decide.
Key Takeaways
- Choose your target precision first; it drives every later decision.
- Calibration data quality is the biggest controllable factor in PTQ outcomes.
- A seven-billion-parameter model quantizes to 4-bit in roughly ten to forty minutes on one GPU.
- Always verify with perplexity, real task evaluation, and manual edge-case checks.
- Benchmark speed and memory on your real target hardware, and deploy behind a flag with rollback ready.