When model output goes wrong, people tend to blame the prompt or the model. Often the real culprit is a sampling setting that was left at a default, copied from a tutorial, or turned in the wrong direction for the wrong reason. Sampling mistakes are sneaky because the output still looks plausible — it just behaves badly in ways that surface later.
This piece names seven specific mistakes we see repeatedly. For each one, you get the symptom, the reason it happens, the cost it imposes, and the corrective practice. The goal is not to memorize rules but to recognize the failure pattern when it shows up in your own work.
None of these require advanced knowledge to fix. Most are matters of habit and attention rather than technique.
Mistake 1: Raising Temperature to Fix Wrong Answers
The most common error is treating temperature as a quality dial. When the model gives a wrong answer, the instinct is to turn temperature up, as if more creativity would find the right one.
Why It Happens and What It Costs
Temperature controls variety, not correctness. Raising it on an inaccurate model spreads the inaccuracy across more diverse answers — now you have several wrong outputs instead of one. The cost is wasted time chasing a fix in the wrong place.
- Corrective practice: if accuracy is the problem, fix the prompt, add context, or switch to a stronger model. Leave temperature out of the accuracy conversation entirely. The foundational guide explains why these are separate concerns.
Mistake 2: Tuning Temperature and Top-p Together
Adjusting both controls in the same experiment feels efficient but makes results impossible to interpret.
Why It Happens and What It Costs
The two controls compound. When you move both and the output changes, you cannot tell which change caused it. The cost is hours of confused experimentation that teaches you nothing reusable.
- Corrective practice: change one control at a time and hold the other at its neutral default. The step-by-step process builds this discipline into the workflow.
Mistake 3: Using High Temperature for Structured Output
Generating JSON, code, or any format with strict rules at a high temperature invites breakage.
Why It Happens and What It Costs
People who do mostly creative work carry a high default into structured tasks out of habit. High temperature lets the model reach for unlikely tokens, which in structured output means malformed syntax, hallucinated fields, or broken brackets. The cost is downstream parsing failures that are hard to trace.
- Corrective practice: drop temperature toward 0 for anything with a rigid format. Determinism is exactly what structured output needs.
Mistake 4: Assuming Temperature 0 Is Fully Reproducible
Teams build tests on the assumption that temperature 0 yields identical output every run, then get flaky failures.
Why It Happens and What It Costs
Temperature 0 makes token selection nearly deterministic, but infrastructure-level nondeterminism can still produce small differences. The cost is brittle tests that fail intermittently and erode trust in the test suite.
- Corrective practice: set a fixed seed where the provider supports it, and write tests that assert on semantic properties rather than exact strings when full reproducibility is not guaranteed.
Mistake 5: Copying Settings From Unrelated Tutorials
A setting that worked beautifully in someone else's blog post gets pasted into a completely different task.
Why It Happens and What It Costs
Numbers look authoritative, so people adopt them without checking whether the underlying task matches theirs. A temperature tuned for fiction lands in a data-extraction pipeline. The cost is output that is subtly wrong for the job in a way nobody investigates.
- Corrective practice: treat any borrowed setting as a starting hypothesis, then run your own sweep. Our examples collection shows how the right setting shifts dramatically by task.
Mistake 6: Never Documenting Your Settings
Settings live in someone's head or scattered across scripts, so the same task gets different settings depending on who runs it.
Why It Happens and What It Costs
Tuning feels like a one-time act, so nobody writes it down. The cost is inconsistent output quality across a team that nobody can explain or fix, because the settings vary invisibly.
- Corrective practice: record the task, temperature, top-p, and prompt version in a shared working checklist so defaults are explicit and reusable.
Mistake 7: Set-and-Forget After a Model Change
Settings tuned for one model version stay untouched after an upgrade, even though the model's behavior changed.
Why It Happens and What It Costs
Once a setting works, it feels finished. But a new model can have a different sensitivity to temperature, so an old default may now be too tame or too wild. The cost is a quiet quality regression that arrives with the upgrade.
- Corrective practice: re-run a quick sweep whenever you change models. The best-practices guide treats model changes as a standing trigger to re-tune.
How These Mistakes Compound
In isolation, each mistake is recoverable. The real damage comes from how they reinforce one another.
The Compounding Chain
Consider a common chain. A team copies a high temperature from a tutorial (Mistake 5). Because the setting is never documented (Mistake 6), nobody questions it. When output comes back wrong, someone raises the temperature further to "improve" it (Mistake 1), which only spreads the inaccuracy. After a model upgrade, the now-doubly-wrong setting is left untouched (Mistake 7). The result is output that is bad for reasons that are now buried under four layers of accumulated error.
Breaking the Chain
The fastest way to break the chain is documentation. A recorded setting with its rationale exposes a copied number as a hypothesis, makes a reflexive increase look suspicious, and flags the setting for review at the next model change. One disciplined practice neutralizes four mistakes at once, which is why the working checklist puts recording front and center.
A Diagnostic Order When Output Goes Wrong
When you hit bad output, the order in which you investigate determines how much time you waste.
Check Cheap Causes First
- Setting: is the temperature appropriate for the task type? This takes seconds and is the most overlooked.
- Prompt: is the instruction clear and constrained? Tighten it before blaming the model.
- Context or model: only after the first two, consider whether the model lacks the information or capability.
Teams that investigate in the reverse order — suspecting the model first — routinely lose days on a problem that a quick look at the setting would have solved.
Frequently Asked Questions
Which of these mistakes is the most damaging?
Raising temperature to fix wrong answers, because it sends you chasing the problem in entirely the wrong place. It wastes the most time and never actually resolves the underlying accuracy issue.
How do I know if a setting was copied from the wrong context?
Run a quick sweep on your own task and compare. If your independently tuned setting lands far from the borrowed one, the borrowed value was tuned for a different kind of work. Always verify rather than assume.
Is it ever fine to leave settings at the default?
Yes, for casual or one-off use where consistency does not matter. The mistakes here bite hardest in repeated, production, or team settings where invisible inconsistency compounds over time.
Why do my temperature 0 tests fail intermittently?
Because temperature 0 is nearly but not perfectly deterministic. Infrastructure nondeterminism can shift outputs slightly. Use a fixed seed where available and assert on meaning rather than exact text.
How often should I re-check my settings?
Re-check whenever you upgrade the model or substantially rewrite the prompt. Outside of those triggers, stable settings rarely need attention. Re-tuning on a calendar is usually unnecessary.
Key Takeaways
- Temperature controls variety, not correctness; never raise it to fix wrong answers.
- Tune one control at a time so you can interpret what changed.
- Use low temperature for structured output, and never assume temperature 0 is perfectly reproducible.
- Treat borrowed settings as hypotheses, document your own, and re-tune after any model change.
- Most sampling mistakes are habits, not technique gaps, so the fixes are about discipline and documentation.