A local language model can feel like it is working while quietly failing in ways you will not notice until something downstream breaks. The output looks plausible, the model responds, and yet latency is creeping up, memory is near its limit, and quality has drifted since the last update. The only way to catch these before they hurt is to measure, and the surprising part is how few measurements you actually need.
This piece defines the small set of metrics that matter for on-device models, explains how to instrument each one without elaborate tooling, and describes how to read the signal each provides. The emphasis is on metrics you can act on. A number you cannot respond to is noise, and there are plenty of impressive-sounding measurements in this space that fall into that bucket.
Group these into three families: speed, resource use, and quality. Each family has one or two metrics that earn their place, and together they give you a complete enough picture to run a local model with confidence.
Speed Metrics: How Fast the Model Responds
Speed is the most visible dimension and the easiest to measure, but the useful version is more specific than a stopwatch on the whole request.
Tokens per second
This is the core throughput metric. It tells you how quickly the model generates output independent of how long the response is.
- Capture it by dividing tokens generated by generation time, which most runtimes report directly.
- Read it against your tolerance: a conversational use case needs higher tokens per second than a batch summarization job.
Time to first token
How long before the model starts responding shapes how the experience feels even more than raw throughput.
- Capture it as the gap between sending the prompt and receiving the first token.
- Read it as a responsiveness signal; high time to first token makes a fast model feel sluggish.
Our overview of self-hosting models explains how runtime configuration influences both of these.
Resource Metrics: Whether the Setup Is Sustainable
A model that runs fast but pegs your memory is one large prompt away from failure. Resource metrics tell you how much headroom you have.
Peak memory use
The high-water mark of RAM or VRAM during inference is what determines whether you are safe or one request from a crash.
- Capture it by monitoring memory during a realistic, large-context request, not an idle one.
- Read it against your total available memory; the gap is your safety margin for larger prompts and concurrency.
Thermal sustained throughput
On laptops especially, the throughput that matters is the one you can sustain, not the one you hit in the first thirty seconds.
- Capture it by running a long session and watching whether tokens per second decays.
- Read it as a sign of thermal throttling if performance drops over time.
The common mistakes practitioners make include trusting a short benchmark that hides exactly this decay.
Quality Metrics: Whether the Output Is Good Enough
Speed and resources mean nothing if the output is wrong. Quality is harder to measure but not impossible, and a little structure beats vibes.
Task success rate
For any repeatable task, the most honest metric is how often the output is acceptable.
- Capture it by running a fixed set of representative prompts and scoring each output pass or fail against clear criteria.
- Read it as your real quality baseline, and rerun the same set after any model or configuration change.
Quality drift after updates
The single most important quality signal is whether your success rate moves when something changes.
- Capture it by rerunning your fixed prompt set after every update.
- Read it as a regression alarm; a drop means the update hurt your specific tasks even if it helped in general.
Our practical examples piece shows how to assemble a representative prompt set worth measuring against.
Reading the Metrics Together
No single metric tells the whole story; the picture emerges from how they move relative to each other.
Common patterns and what they mean
- High tokens per second but rising peak memory means you are fast now but fragile under larger prompts.
- Good throughput that decays over a long session points to thermal throttling, not a model problem.
- Stable speed and memory but falling task success means an update regressed your tasks, and the fix is rollback, not tuning.
The decision framework for local deployments places this monitoring inside its ongoing maintenance stage.
Building a baseline you can trust
A measurement only means something against a baseline, and the baseline has to be captured under honest conditions. Take your readings on realistic prompts at realistic load, not on a trivial test that flatters the numbers. Record the model version and settings alongside the figures, because a metric without that context is impossible to interpret later. Once you have a baseline, the job shifts from measuring absolutes to watching for movement, and movement is where the actionable signal lives.
The discipline that ties this together is rerunning the same measurements the same way every time. Comparing today's tokens per second against a number captured under different conditions tells you nothing. Consistency in how you measure is what turns a pile of numbers into a trend you can act on.
Metrics Worth Ignoring
Part of measuring well is refusing to measure things that do not help. Several impressive-sounding numbers add noise without adding insight.
What to leave out
- Aggregate quality scores with no task tie-in. A single blended quality number that does not map to a real task you care about tells you little about whether the model is useful for your work.
- Idle resource readings. Memory and throughput measured while the model sits idle bear no relation to its behavior under a real request, yet they are the easiest numbers to capture.
- Peak throughput from a cold, short burst. The first few seconds of generation often look better than the sustained rate, especially before thermal limits engage, so a brief burst overstates real performance.
Cutting these keeps your attention on the handful of metrics you can actually respond to, which is the entire point of measuring at all.
Turning Metrics Into Decisions
Measuring is only half the job; the other half is having a decision ready for each signal so the numbers actually change what you do. A metric you watch but never act on is a slightly more sophisticated form of ignoring the problem.
Pairing each signal with an action
- Falling task success after an update triggers a rollback, not a tuning session, because the cause is the update, not your configuration.
- Rising peak memory triggers a smaller context window or model, before the next large prompt crashes the setup.
- Decaying throughput over a long session triggers a thermal investigation, since the problem is sustained load, not the model.
- High time to first token on an interactive use case triggers a configuration review, because responsiveness is the experience users feel.
The value of pre-pairing signals with actions is that you respond quickly and correctly under pressure instead of improvising when something is already failing. A measurement program with no decisions attached generates dashboards no one reads; one with a clear action per signal becomes the early-warning system that lets you run a local model with genuine confidence.
Frequently Asked Questions
Which metric should I track first?
Task success rate, because it is the one that actually maps to whether the model is useful. Speed and memory matter, but a fast, efficient model that produces wrong answers is worthless. Start with a fixed prompt set and a clear pass-fail standard.
Do I need special tooling to capture these?
No. Most runtimes report tokens per second and time to first token directly, system tools show memory use, and task success is a manual scoring pass. Elaborate observability stacks are optional for personal and small-team use.
How often should I rerun my quality measurements?
After every model update or significant configuration change. Quality drift is the metric most likely to bite silently, and the only way to catch it is to rerun the same prompts and compare success rates.
Why measure time to first token separately from throughput?
Because they shape the experience differently. A high time to first token makes even a fast model feel unresponsive, since the user waits before seeing anything. For interactive use, it can matter more than raw tokens per second.
What does a sudden memory spike usually indicate?
Usually a larger-than-typical prompt pushing context use up, or concurrent requests stacking. Measuring peak memory on a realistic large request rather than an idle one is how you see this coming before it crashes the model.
Key Takeaways
- A small set of metrics across speed, resources, and quality is enough to run a local model confidently.
- Tokens per second and time to first token capture different aspects of how fast the model feels.
- Peak memory on a realistic large request reveals your true safety margin.
- Task success rate against a fixed prompt set is the most honest quality metric.
- Rerun quality measurements after every update, because drift is the failure that hides best.