You understand what weights are. Now you want to actually do something with them: download a model, look inside the file, shrink it to fit your hardware, and adapt it to your task. This guide is the sequential process for exactly that, with the decision you face at each step spelled out.
We assume you have read at least the Beginner's Guide so the vocabulary is familiar. Follow these steps in order; each one builds on the last, and skipping the early ones causes the most common headaches later.
The whole path is: pick a model, verify and load its weights, check that it fits your memory, quantize if it does not, then optionally fine-tune. Do not jump to fine-tuning before you can reliably load and run the base model.
Step 1: Choose a Model by Size and Format
Before touching weights, decide which model and which file you are pulling. Two attributes matter most.
- Parameter count sets your memory budget. As a rule, multiply parameters in billions by 2 to estimate gigabytes needed at 16-bit precision. A 7B model needs roughly 14 GB.
- File format affects safety and tooling. Prefer safetensors over legacy
.binfiles, because safetensors cannot execute code when loaded.
Pick the smallest model that plausibly does your job. Oversizing is the most common and most expensive early mistake, which the Common Mistakes article covers in depth.
Step 2: Verify the Weights Before Loading
Never load a weight file you have not verified. Two checks take seconds and prevent real problems.
- Checksum. Compare the published hash against the file you downloaded so you know it is complete and untampered.
- Format. Confirm the file is safetensors or another non-executable format. If you must use a pickle-based file, only do so from a source you fully trust.
This step exists because weight files are arbitrary binary data, and the older formats can run code on load. Treat downloaded weights like any other executable from the internet.
Step 3: Load the Weights and Confirm the Shape
Load the model and immediately inspect it before running inference. You are confirming the weights match the architecture you expected.
- Print the total parameter count and compare it to the advertised number. A mismatch means you grabbed the wrong file.
- List a few layer names and their tensor shapes. This tells you the model loaded with the right structure.
- Run one tiny test prompt. If it produces coherent output, the weights are intact and correctly mapped.
If the model loads but produces gibberish, the usual culprit is a precision or architecture mismatch between the weights and the loading code, not corrupt weights.
Step 4: Measure Memory and Decide on Quantization
Now check whether the model fits your hardware as loaded.
If it fits
Run at the native precision, typically 16-bit. You get full quality and the simplest setup. Confirm memory headroom for the context window, which also consumes memory that grows with input length.
If it does not fit
Quantize. This converts each weight to fewer bits, shrinking memory use.
- 8-bit roughly halves memory with minimal quality loss. Try this first.
- 4-bit roughly quarters memory with a small but real quality drop. Use it when 8-bit still will not fit.
Quantize to the largest precision that fits, not the smallest available. Going lower than necessary throws away quality for no reason. The Best Practices guide explains how to measure the quality cost so you choose deliberately.
Step 5: Establish a Baseline
Before fine-tuning anything, measure the base model on your actual task. Write 15 to 30 representative test cases and record how the unmodified model does.
This baseline is non-negotiable. Without it you cannot tell whether fine-tuning helped, hurt, or did nothing. Many teams skip this and end up unable to justify the time they spent adjusting weights.
Step 6: Fine-Tune the Weights (If Needed)
Only fine-tune if the base model genuinely falls short after good prompting. When you do, choose the efficient path first.
- Prepare data. Assemble a clean, consistent dataset of input-output pairs in your target style. Quality beats quantity; a few hundred excellent examples often beat thousands of noisy ones.
- Use LoRA. Parameter-efficient fine-tuning freezes the original weights and trains a small adapter. It runs on a single GPU and produces a small file you can swap in and out.
- Set a conservative learning rate. Too high and the weights overshoot, erasing general ability. Start low and watch the loss.
- Validate against your held-out cases. Compare to the Step 5 baseline. Keep the adapter only if it measurably wins.
Reserve full fine-tuning, which updates every weight, for cases where LoRA proves insufficient. It is far more expensive and risks catastrophic forgetting.
Step 7: Save, Version, and Document
Treat your final weights like code.
- Save adapters and merged models with clear version names.
- Record the base model, quantization level, dataset, and learning rate that produced them.
- Keep the checksum of what you shipped so you can verify it later.
This discipline lets you reproduce results and roll back when an update regresses. The Checklist turns these steps into a reusable working tool.
Common Stumbles and How to Recover
Even with the sequence above, a few problems recur often enough to name. Knowing the recovery for each saves hours.
- Gibberish after loading. Almost always a configuration mismatch, not corrupt weights. Reload with the exact precision and tokenizer the model card specifies before suspecting the file.
- Out-of-memory on long inputs. You budgeted for weights but not the context window. Reduce the maximum context, quantize one level further, or move to hardware with more memory.
- Fine-tune made things worse. Usually too high a learning rate or too many steps. Lower the rate, cut the steps, and re-validate against your Step 5 baseline.
- Quantized model fails on hard cases only. You went a level too low. Move from 4-bit back to 8-bit if your hardware allows, and re-measure on the cases that broke.
The thread connecting all four is that the diagnosis comes from your baseline and your test cases. Without those, each of these stumbles becomes a guessing game; with them, the fix is usually obvious within a few minutes. This is why Steps 5 and the verification in Step 2 matter as much as the flashier work of fine-tuning.
Frequently Asked Questions
How do I estimate how much memory a model needs?
A quick rule is parameters in billions times 2 for 16-bit precision, so a 7B model needs about 14 GB. For 8-bit, multiply by 1; for 4-bit, by about 0.5. Add extra headroom for the context window, which grows with input length and is easy to forget.
Should I quantize before or after fine-tuning?
Establish your workflow on the unquantized or lightly quantized model first, then fine-tune, then quantize for deployment if needed. Some methods let you fine-tune on top of a quantized base to save memory, but starting at higher precision gives you a cleaner baseline to compare against.
When is fine-tuning actually worth it?
Fine-tune only after good prompting and retrieval fall short on your specific task. If the base model already does the job with the right instructions, adjusting weights adds cost and maintenance for no benefit. Always compare against a measured baseline before committing.
Why does my loaded model output gibberish?
The most common cause is a mismatch between the weights and the loading code, such as wrong precision, a wrong tokenizer, or an architecture that does not match the file. Corrupt downloads are less common if you checksum. Reload with the exact configuration the model card specifies.
What is the difference between LoRA and full fine-tuning?
Full fine-tuning updates every weight in the model, which is powerful but expensive and risks erasing general ability. LoRA freezes the original weights and trains a small set of new ones, running on modest hardware and producing a tiny swappable file. LoRA is the right default for most teams.
Key Takeaways
- Pick the smallest viable model and prefer safetensors before you download anything.
- Always verify weights with a checksum and confirm the format before loading.
- Inspect parameter count and shapes on load, and run a tiny test before trusting the model.
- Quantize to the largest precision that fits your hardware, never lower than necessary.
- Establish a measured baseline before fine-tuning, prefer LoRA, and version your final weights like code.