Overfitting and underfitting are not vibes you eyeball on a loss curve. They are measurable gaps between how a model performs on the data it learned from and how it performs on data it has never seen. If you cannot put a number on that gap, you cannot tell a stakeholder whether a model is ready, and you cannot tell an engineer what to fix next.
The trap is that a single metric — say, accuracy — hides both failures at once. A model that is 99% accurate on training data and 71% accurate on validation data is overfit. A model that is 68% on both is underfit. Same validation score range, completely different disease, completely different cure. The right instrumentation makes the diagnosis obvious before you ship.
This piece is about the specific metrics that separate the two conditions, how to instrument them so the signal is trustworthy, and how to read the numbers without fooling yourself. If you want the conceptual grounding first, start with The Complete Guide to Ai Model Overfitting and Underfitting.
The Generalization Gap Is the Master Metric
Everything else is a refinement of one number: the gap between training performance and held-out performance.
How to Compute It
Pick your primary metric (accuracy, F1, AUC, RMSE — whatever maps to the business outcome). Measure it on the training set and on a validation set the model never touched. The generalization gap is the difference.
- Large gap, high train score: overfitting. The model memorized.
- Small gap, low scores on both: underfitting. The model lacks capacity or signal.
- Small gap, high scores on both: the goal. Generalizing.
Why It Beats a Single Score
A lone validation number tells you how good but not why it is not better. The gap tells you the direction to move. If the gap is wide, you reduce capacity or add regularization. If both scores are low and close, you add capacity, features, or data quality. You cannot derive that from one number.
Track the Train/Validation Curve, Not Just the Endpoint
A point-in-time score is a snapshot. The shape over training epochs is the diagnosis.
Reading Learning Curves
Plot training and validation loss against epochs (or training set size). Three patterns recur:
- Diverging curves: training loss keeps dropping while validation loss bottoms out and rises. Classic overfitting. The inflection point is where you should have stopped.
- Both flat and high: the model plateaus early at poor performance on both. Underfitting — more epochs will not help.
- Both converging low: healthy.
Learning Curves vs Training Set Size
Vary the amount of training data and re-plot. If validation performance is still climbing as you add data, more data will help. If it flattened long ago, you have a capacity or feature problem, not a data-volume problem. This distinction saves teams from buying expensive labeling they do not need.
Metrics That Expose Overfitting Specifically
When the gap is wide, these sharpen the picture.
- Cross-validation variance: run k-fold CV and look at the standard deviation across folds. High variance between folds means the model is sensitive to which examples it saw — an overfitting signature.
- Performance on perturbed inputs: small input noise that craters accuracy signals brittle, memorized boundaries.
- Confidence calibration: overfit models are often overconfident. Compare predicted probability to actual frequency (a reliability diagram or Expected Calibration Error). Confident and wrong is the overfitting tell.
For the hands-on fixes once you confirm overfitting, A Step-by-Step Approach to Ai Model Overfitting and Underfitting walks through the remediation order.
Metrics That Expose Underfitting Specifically
A narrow gap with low scores points the other way.
- Training error itself: if the model cannot even fit the training set well, capacity or features are the bottleneck. This is the cleanest underfitting signal — it needs no held-out data.
- Residual structure (regression): plot residuals against predictions. Visible patterns mean the model is missing a relationship it could be capturing.
- Bias measured against a stronger baseline: train a deliberately larger model on the same data. If it blows past your current model on training error, your original was underfit.
Instrumenting So the Signal Is Trustworthy
The most common reason these metrics mislead is contaminated measurement, not bad math.
Hold the Test Set Sacred
Use three splits: train, validation, test. You tune against validation. You touch test once, at the end. Every time you peek at the test set and adjust, you leak information and inflate your numbers. A test score you optimized against is no longer a measure of generalization — it is a measure of how hard you cheated.
Prevent Leakage
Leakage manufactures fake high scores that look like great generalization until production. Guard against:
- Fitting scalers or encoders on the full dataset before splitting (fit on train only).
- Time-series splits that let future data into training — always split chronologically for temporal data.
- Duplicate or near-duplicate rows straddling the split.
Match the Metric to the Decision
Accuracy on imbalanced data is a vanity metric. If 95% of cases are negative, a model that predicts "negative" always scores 95% and detects nothing. Use precision, recall, F1, or AUC depending on the cost of each error type. The best practices guide covers metric selection by problem type in more depth.
Choosing a Primary Metric Before You Measure Anything
A subtle failure precedes all the others: measuring the wrong thing well. Before you compute a single gap, decide which metric maps to the decision the model drives, and let everything else be secondary diagnostics.
Match the Metric to the Cost of Errors
- False negatives expensive (fraud, disease screening): optimize recall; a missed positive is the costly error.
- False positives expensive (spam filters, irreversible actions): optimize precision; a false alarm is the costly error.
- Both matter, classes imbalanced: use F1 or AUC rather than accuracy, which a majority-class predictor games.
- Probabilities drive downstream logic: add calibration to your primary set, not as an afterthought.
Keep One Primary, Several Diagnostic
Pick one primary metric that the business cares about and report it as the headline gap. Keep the others — calibration, per-segment scores, cross-validation variance — as diagnostics that explain why the primary metric is where it is. Reporting ten co-equal numbers obscures the decision; one headline plus supporting diagnostics clarifies it.
A Practical Measurement Routine
Run this on every model before you call it done.
- Confirm clean splits and no leakage.
- Record train and validation scores on your primary metric; compute the gap.
- Plot learning curves over epochs and over data size.
- Run k-fold CV; record mean and standard deviation.
- Check calibration if probabilities drive decisions.
- Touch the test set exactly once for the final reported number.
If the gap is wide, you have an overfitting problem to regularize. If both scores are low and close, you have an underfitting problem to add capacity to. The numbers, not your intuition, tell you which.
Frequently Asked Questions
What generalization gap is "too big"?
There is no universal threshold — it depends on baseline difficulty and stakes. A useful rule: if the gap is more than a few percentage points and growing as you train, treat it as overfitting worth addressing. Compare against a simple baseline model's gap rather than an absolute number.
Can a model overfit and underfit at the same time?
Not on the same metric, but a model can underfit some patterns while overfitting noise in others — common with mixed-quality features. Per-segment metrics reveal this: strong on one slice, memorized on another. Segmented evaluation exposes it.
Is high training accuracy always bad?
No. High training accuracy is only a problem when validation accuracy lags far behind it. High training and high validation accuracy together is exactly what you want. The gap is what matters, not the training number in isolation.
Why use cross-validation instead of one validation split?
A single split can be lucky or unlucky depending on which examples land in it. K-fold cross-validation averages over multiple splits and, critically, reports the variance — and high fold-to-fold variance is itself an overfitting signal.
Which metric should I optimize for imbalanced classes?
Avoid raw accuracy. Use precision and recall (or F1 to balance them) and AUC for ranking quality. Choose based on which error is more expensive: recall when missing positives is costly, precision when false alarms are.
Key Takeaways
- The generalization gap — train score minus held-out score — is the master metric; it tells you direction, not just quality.
- Learning curves over epochs and over data size separate "needs more data" from "needs more capacity."
- Wide gap plus high train score equals overfitting; low-and-close scores equal underfitting.
- High cross-validation variance, overconfidence, and brittleness under perturbation are overfitting fingerprints.
- Protect measurement integrity first: clean splits, no leakage, a test set touched once, and a metric matched to the actual decision.