Principles are easy to nod along to and hard to apply until you have seen them in motion. This guide walks through five concrete tasks where the question "how sure is the model, really?" mattered, shows the prompts that surfaced honest uncertainty, and explains what made each example succeed or fall flat. The scenarios are deliberately ordinary — the kind of work people do with models every day.
In each case the model started out sounding equally confident about everything. The interesting part is watching how a small change in the prompt separated the claims it could support from the ones it was inventing. Where an example failed, the failure is just as instructive as the success, because it shows the limits of what prompting alone can fix.
Read these alongside the underlying practices if you want the why behind each move. Here the focus is the what: specific tasks, specific prompts, specific outcomes you can copy and adapt.
Example 1: Summarizing a Dense Contract
A team fed a model a services agreement and asked for the key terms. The first pass listed a liability cap, a termination notice period, and a renewal clause — all stated as flat fact.
What went wrong at first
One of the three "facts" was wrong: the model had inferred a renewal clause that the contract did not contain. It sounded identical in confidence to the two correct items.
The prompt that fixed it
"List each term. For each, quote the exact contract language it comes from, or mark it as 'not found in text' and label it low confidence."
Forcing a quote per claim collapsed the fabricated renewal clause to "not found." Grounding confidence in traceable text — the practice from the best practices guide — is what carried this.
Example 2: Answering a Medical-Adjacent Question
A user asked a general assistant about a drug interaction. The default answer was a confident, specific claim.
Why confidence here is dangerous
Stakes are high and the model has no way to verify against current clinical sources. A confident wrong answer could cause harm.
The prompt that helped
"If this requires up-to-date or clinical information you cannot verify, say so plainly and recommend a professional source. Only state interactions you are highly confident are established."
The model shifted to flagging uncertainty and pointing to a pharmacist. Note the limit: this makes the answer honest, not complete. Prompting cannot give the model knowledge it lacks — a point the trade-offs guide treats directly.
Example 3: Debugging a Code Snippet
A developer pasted a function and asked why it failed. The model proposed a fix with total confidence.
The trap
The suggested fix did not run. The model had pattern-matched a plausible-looking correction without executing it.
The prompt that calibrated it
"Walk through the code line by line. Mark any fix you have not mentally traced through as a hypothesis, not a solution, and say what test would confirm it."
This split "I'm sure" from "I think." The developer ran the flagged hypothesis, confirmed it, and saved time chasing the wrong lead. For code, ground truth is execution — see how the how-to process builds a test set around that.
Example 4: Drafting a Factual Briefing
A researcher asked for a briefing on a niche historical event. The draft mixed well-documented facts with confident-sounding but invented dates.
Why it failed silently
The invented details were woven seamlessly into the true ones. Nothing in the tone distinguished them.
The prompt that separated them
"Tag each sentence as 'well-documented,' 'commonly stated but I'm unsure,' or 'I am inferring this.' Do not present inferences as documented facts."
The briefing came back with the soft claims visibly tagged. The researcher verified only the tagged items, cutting checking time sharply. This is the granularity the common mistakes guide argues for.
Example 5: Forecasting From Ambiguous Data
An analyst asked a model to project a trend from a short, noisy dataset. The first answer gave a single confident number.
The problem with false precision
The data did not support a point estimate. A confident single figure implied certainty the situation could not justify.
The prompt that calibrated it
"Given how little data this is, give a range rather than a point, state your assumptions, and rate your confidence in the projection as low, medium, or high with a reason."
The model returned a range, named its assumptions, and rated the projection low confidence — an honest answer that prevented the analyst from over-committing to a fragile number.
Example 6: Triaging a Pile of Inbound Messages
A small team used a model to sort inbound messages into categories like billing, bug report, and feature request. The model assigned every message a category with no hesitation.
Where it broke down
Genuinely ambiguous messages — ones that touched two categories or were too vague to classify — got forced into a single bucket with the same confidence as the clear-cut ones. The team only discovered the misroutes after the fact, when messages landed with the wrong queue.
The prompt that calibrated it
"Assign a category only if you are confident. If a message is ambiguous or spans categories, label it 'needs human triage' and say why. Add a confidence level to every assignment."
The model began routing the clear messages automatically and flagging the genuinely ambiguous ones for a person. The team's misroute rate fell because the model stopped pretending the hard cases were easy. This mirrors the case study, where letting the model escalate the uncertain cases was the whole unlock.
What the Failures Have in Common
Looking across the cases that went wrong before calibration, a single pattern repeats.
Uniform confidence on non-uniform certainty
In every failed first pass, the model applied the same confident tone to claims of wildly different reliability — a quoted fact and an invented one, a traced fix and a guessed one, a clear category and an ambiguous one. The damage came not from the model being wrong, but from it hiding which parts were wrong.
The fix is always differentiation
The calibration prompt's job, in every example, was to make the model differentiate: to mark the soft claims so a human could find them. None of these prompts made the model smarter. They made it honest about the distribution of its own certainty, which is what let people trust the confident parts and check the rest. That is the same lesson the best practices guide builds its rules around.
Frequently Asked Questions
What is the common thread across these examples?
Each one separates claims the model can support from claims it is inventing, and ties confidence to evidence rather than tone. Whether it is quoting contract text, tracing code, tagging documented facts, or giving a range instead of a point, the move is the same: force the model to show its basis so its confidence reflects support rather than fluency.
Did prompting fix every problem in these scenarios?
No, and that is important. In the medical example, prompting made the answer honest but could not supply knowledge the model lacked. Calibration through prompts reliably surfaces uncertainty and prevents confident fabrication; it does not turn a model into an authoritative source on things it genuinely does not know.
Why quote the source text in the contract example?
Because requiring an exact quote per claim forces the model to ground each statement in real evidence, which collapses any claim it cannot trace. The fabricated renewal clause could not produce a quote, so it surfaced as "not found." Grounding confidence in traceable text is far stronger than asking the model to rate itself on feel.
How is calibrating code different from calibrating facts?
For code, the ground truth is execution — does it run and pass tests — so calibration means having the model flag fixes it has not actually traced as hypotheses rather than solutions. You then verify by running the flagged item. With facts, ground truth is documentation, so the model tags claims by how well-documented they are.
Why ask for a range instead of a single number in forecasts?
Because a single confident number implies precision that noisy or sparse data cannot support. A range, paired with stated assumptions and a confidence rating, honestly communicates the uncertainty in the projection. It stops a decision-maker from over-committing to a fragile point estimate that merely sounded authoritative.
Can I reuse these exact prompts on my own tasks?
Yes, as starting points. Adapt the grounding instruction to your evidence source — contract text, code execution, documentation, or data — and keep the core moves: require a basis for each claim, reason before rating, and allow an honest exit. Then validate on a small test set, because calibration is specific to your model and domain.
Key Takeaways
- Across tasks, the winning move is forcing the model to show its basis so confidence reflects evidence, not tone.
- Requiring an exact source quote per claim collapses fabrications that have no support.
- For code, treat unexecuted fixes as hypotheses and verify by running them.
- Tagging each claim by how well-documented it is lets you verify only the soft parts.
- For forecasts, ask for a range with stated assumptions instead of a falsely precise point estimate.
- Prompting makes answers honest about uncertainty; it cannot supply knowledge the model genuinely lacks.