A forecast that is never scored is a forecast you are taking on faith. The whole point of metrics is to replace that faith with evidence: a small set of numbers that tell you whether your forecasting tool is earning its place or quietly leading you astray. The trouble is that the obvious metric, "how close was the number," is the least informative one, and teams that stop there miss the failures that matter most.
This piece defines the KPIs worth tracking, explains how to instrument each one, and, most importantly, shows how to read the signal each produces. A metric you cannot interpret is just another number on a dashboard. The aim is to make every metric here actionable: when it moves, you should know what to do.
Group these into three families: accuracy, calibration, and stability. You need all three, because each catches a failure the others miss.
Accuracy: How Far Off Was the Forecast
Accuracy metrics measure the gap between forecast and actual. They are necessary but, on their own, dangerously incomplete.
What to track
- Mean absolute percentage error. The average size of your misses as a percentage, easy to explain to leadership.
- Bias, or mean error. Whether you systematically over- or under-forecast. A small average error can hide a consistent directional bias.
How to read it
A rising error trend signals drift. A persistent bias signals a broken assumption, not random noise, and points you toward a specific driver to investigate. The failures these metrics expose are catalogued in Seven Ways Forecasting Models Quietly Mislead Finance Teams.
Calibration: Was the Uncertainty Honest
This is the metric family most teams skip, and it is the one that separates a trustworthy forecast from an overconfident one.
What to track
- Interval coverage. How often actuals fall inside your stated prediction interval. A ninety percent interval should contain actuals about ninety percent of the time.
How to read it
If actuals land inside your interval far less often than promised, your forecast is overconfident and your stated range is a lie. If they land inside far more often, your interval is too wide to be useful. Honest calibration is what made the difference in When a SaaS Finance Team Rebuilt Its Forecast With AI.
Stability: Is the Forecast Behaving Consistently
Stability metrics catch a forecast that thrashes, swinging wildly from period to period even when the business does not.
What to track
- Forecast revision volatility. How much your forecast for a fixed future period changes as you re-run it each cycle.
How to read it
A forecast that lurches each month is hard to plan against and usually signals an over-reactive model or noisy input data. Some revision is healthy as new information arrives, but large unexplained swings are a warning. Stable, gradually updating forecasts are a sign of the discipline described in Disciplines That Keep an AI Forecast Trustworthy.
How to Instrument These Metrics
Metrics are worthless if you compute them once and forget them. Build them into the rhythm of your close.
The practice
- Snapshot every forecast. Store each forecast with its date and interval so you can later compare it to actuals. Without snapshots, you cannot compute anything historically.
- Score at every close. When actuals land, compute accuracy, bias, and interval coverage for the period that just ended.
- Track trends, not single points. One period's error tells you little. The trend across periods is where drift and bias become visible.
This instrumentation is the Detect stage of The Drift-Decompose-Decide Loop for Smarter Forecasts.
Reading the Whole Dashboard Together
No single metric tells the story. A forecast can be accurate on average while badly biased, well calibrated while thrashing, or stable while consistently wrong. Read the three families together.
The healthiest pattern is low and stable error, honest interval coverage near its stated level, and gradual revisions that track new information. When any one of the three degrades while the others hold, you have a specific, diagnosable problem rather than a vague sense that something is off. That specificity is the entire value of measuring.
Setting Thresholds Without Fooling Yourself
Resist the urge to set a single accuracy target and call any miss a failure. A more honest approach defines an acceptable error band based on your historical backtest, then watches for the trend leaving that band. The band acknowledges that forecasting is probabilistic; the trend monitoring catches real degradation. Pair this with the gating logic in Pre-Launch Checks Before You Trust an AI Forecast.
Segment Your Metrics, Don't Just Aggregate
A single company-wide accuracy number can hide a forecast that is excellent for one product line and disastrous for another. The two errors cancel in the aggregate, leaving a comfortable average that masks a real problem. The fix is to compute your core metrics by segment, whether that means product line, region, or customer cohort, wherever the forecast actually drives separate decisions.
Why segmentation pays off
Segmented metrics turn a vague "the forecast is a little off" into a precise "we are systematically over-forecasting the enterprise segment." That precision is what makes a metric actionable. It points to a specific driver and a specific assumption to investigate, rather than leaving you to guess where the aggregate error came from. The cost is a busier dashboard, which is a small price for the diagnostic power, and it mirrors the decomposition discipline in The Drift-Decompose-Decide Loop for Smarter Forecasts.
Tie Each Metric to an Action
A metric that does not change what you do is decoration. Before adding any KPI to your dashboard, decide in advance what a bad reading will trigger. Rising error trend triggers a re-decomposition of the affected driver. Poor interval coverage triggers a recalibration of the model's uncertainty. High revision volatility triggers an investigation of the input data feeding the forecast. Persistent bias triggers a hunt for the broken assumption behind the consistent direction.
When every metric has a predefined response, the dashboard stops being a wall of numbers and becomes a control panel. This is the discipline that separates teams who measure for reassurance from teams who measure to act, and it is the same accountability that runs through Disciplines That Keep an AI Forecast Trustworthy.
Beware Metrics That Reward the Wrong Behavior
A metric is also an incentive, and a poorly chosen one quietly distorts behavior. If you judge a forecasting team solely on minimizing accuracy error, you create pressure to narrow the prediction interval, because a tight interval looks more impressive even though it is less honest. You can optimize an accuracy number into a forecast that is precise, confident, and frequently wrong about its own uncertainty.
The defense is to never track accuracy in isolation. Pair it with calibration so that narrowing the interval dishonestly shows up immediately as poor coverage. The two metrics check each other: accuracy pushes for a sharp central estimate, calibration insists the uncertainty around it stay honest. Watched together, they prevent the gaming that either one alone invites. This is why the three families described above are designed to be read as a set, not cherry-picked one at a time.
Keep the metric set small
Resist the urge to track everything measurable. A dashboard with twenty metrics gets read like a dashboard with none, because nobody can hold twenty signals in their head or act on all of them. A tight set, one accuracy measure, one bias measure, one calibration measure, and one stability measure, covers the failure modes that matter and stays small enough that each number genuinely gets watched. The goal is not maximal measurement but actionable measurement, which is also the spirit of the gate in Pre-Launch Checks Before You Trust an AI Forecast.
Frequently Asked Questions
Why isn't accuracy the most important metric?
Because a forecast can be accurate on average while systematically biased or wildly overconfident. Accuracy alone hides the failures that calibration and stability catch.
What is interval coverage and why does it matter?
It measures how often actuals fall inside your stated prediction interval. It tells you whether your uncertainty is honest, which is the most decision-relevant thing a forecast communicates.
How often should I compute these metrics?
At every close, scoring the period that just ended, and then reading the trend across periods rather than reacting to any single point.
What does high revision volatility indicate?
A forecast that swings wildly from period to period, usually signaling an over-reactive model or noisy inputs. It makes the forecast hard to plan against.
Should I set a hard accuracy target?
No. Define an acceptable error band from your backtest and watch for the trend leaving it. A single target turns normal variance into false failures.
What does the healthiest dashboard look like?
Low, stable error; honest interval coverage near its stated level; and gradual revisions that track new information rather than thrash.
Key Takeaways
- Metrics replace faith with evidence; the obvious "how close was the number" metric is the least informative on its own.
- Track three families: accuracy, calibration, and stability, because each catches a failure the others miss.
- Interval coverage is the most decision-relevant metric and the one teams most often skip.
- Instrument by snapshotting every forecast, scoring at each close, and reading trends rather than single points.
- Set an acceptable error band from your backtest and watch the trend, rather than treating any single miss as a failure.