Tokens cost money. Context windows determine what your model can "see." Together, they set the ceiling on what your AI workflows can accomplish and the floor on what they'll cost. Yet most teams deploying language models track neither with any rigor—they check the invoice at the end of the month, wince, and move on.
That cycle breaks the moment you treat token usage and context utilization as first-class operational metrics. When you can see exactly how many tokens each workflow consumes, how much of the available context window each request fills, and where the cost and quality cliffs are, you gain the leverage to optimize deliberately rather than accidentally. This article defines the metrics that matter, shows you how to instrument your stack to capture them, and tells you how to interpret the signals once data starts flowing.
Before diving into measurement, it helps to have a working mental model. A token is roughly 0.75 words of English text—a 1,000-word article is approximately 1,300 tokens. A context window is the maximum tokens a model can process in a single forward pass: both the input you send and the output it generates count against that limit. If you're new to those fundamentals, Getting Started with Tokens and Context Windows covers the ground floor before you build on it.
The Right KPIs: What to Actually Track
Most teams measure too little (just total spend) or the wrong things (raw token counts without normalization). The following six metrics give you a complete picture without requiring a data engineering team to maintain.
1. Cost Per Task (CPT)
Total tokens multiplied by per-token price, divided by the number of completed tasks. "Task" can be a summarization, a draft, a customer reply—whatever your workflow produces. CPT is your unit economics metric. A $0.003 CPT on a document summarizer is a business insight; "we spent $47 this week" is not.
Calculate it at the workflow level, not the model level. The same model can have a CPT of $0.001 for a classification task and $0.02 for a long-form draft.
2. Context Utilization Rate (CUR)
CUR = tokens_used / context_window_limitExpressed as a percentage, this tells you how efficiently you're using the available window. A CUR consistently below 20% suggests you're over-provisioning (paying for a large-window model when a smaller one would do). A CUR consistently above 85% is a warning sign: you're approaching truncation risk, and quality may degrade near the edges of the window.
Target range for most workflows: 40–75%. You want headroom, but not wasteland.
3. Input-to-Output Token Ratio
Divide input tokens by output tokens for each request type. Summarization tasks typically run 10:1 to 30:1 (long input, short output). Creative generation runs closer to 2:1 or even 1:1. Chat applications vary widely.
This ratio matters because input and output tokens are priced differently on most APIs—output tokens often cost two to four times more than input tokens. A workflow that produces unexpectedly long outputs is a budget leak you can identify only if you track this ratio separately.
4. Prompt Efficiency Score (PES)
This is a compound metric you calculate yourself:
PES = (output_quality_score × output_tokens) / total_tokens_consumedOutput quality score is whatever rubric your team uses—human rating, automated evaluation against a rubric, task completion rate. PES rewards prompts that produce high-quality outputs without burning tokens to get there. It penalizes bloated system prompts, over-specified instructions, and context stuffing.
5. Truncation Rate
The percentage of requests where the model hits the context window limit and truncates either the input or the output. Even a 1–2% truncation rate on a production workflow means hundreds or thousands of degraded responses per week. Most teams discover this only through downstream errors; measuring it directly lets you act before users complain.
6. Token Velocity
Tokens processed per unit of time across your entire deployment. Token velocity captures throughput demand and is the leading indicator for rate-limit collisions, latency spikes, and scaling decisions. If velocity spikes 3Ă— on Monday mornings, you can pre-warm, batch, or negotiate higher rate limits before the problem surfaces as user-facing slowness.
Instrumentation: How to Capture These Metrics
Defining metrics is pointless without a collection layer. Here's a practical instrumentation stack that works whether you're a solo operator or running agency-scale deployments.
Intercept at the API Call
Every major LLM provider returns token usage in the response object. OpenAI's API, for example, returns usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens in every response. Anthropic's API returns similar fields. Log these fields to your datastore on every call—not just failures, not just samples. Sampling gives you averages; complete logging gives you tail behavior, which is where problems live.
Tag Every Request
Each log entry should carry:
- Workflow ID — which pipeline made the call
- Task type — summarization, classification, generation, etc.
- Model name and version — costs and capabilities change across versions
- Environment — production, staging, evaluation
- User or account ID — for multi-tenant applications
Without tags, you have usage data. With tags, you have actionable segmentation.
Calculate CUR at Log Time
Don't rely on inference. Pull the model's context window limit from a configuration file or lookup table and compute CUR immediately when you log the response. Storing raw token counts and context limits separately means you'll reconstruct this calculation a dozen times in twelve different queries. Compute once, store the result.
Aggregate Into a Dashboard
A simple Postgres table with daily rollups is sufficient for most teams. You want to see, at minimum:
- CPT by workflow, trended over 30 days
- CUR distribution (histogram, not just average—averages hide bimodal patterns)
- Truncation rate by task type
- Input/output ratio by workflow
Tools like Metabase, Grafana, or even a well-structured spreadsheet work fine. You don't need a purpose-built LLM observability platform until you're running hundreds of thousands of calls per day. Platforms like Langfuse, Helicone, and Weights & Biases Prompts can accelerate this if you're scaling fast, but the underlying metrics are the same.
Reading the Signal: What the Numbers Tell You
Data without interpretation is noise. Here's how to translate each metric into a decision.
High CPT on a Specific Workflow
Audit the system prompt first. Teams routinely ship system prompts that are 800–1,200 tokens long when 200 tokens would do the same job. Every unnecessary token in a system prompt costs you on every single call. Strip it to the minimum and measure quality before and after. In most cases, leaner prompts perform equivalently and occasionally outperform bloated ones because the model isn't wading through noise.
CUR Consistently Below 30%
You're likely using a large-window model for tasks that don't require it. A workflow that averages 4,000 tokens doesn't need a 128K context window—and large-window models typically cost more per token than their smaller-window counterparts. Downgrading to a right-sized model is a straightforward cost lever. The ROI of Tokens and Context Windows walks through the math for making this case internally.
CUR Spiking Above 85%
You're at truncation risk. The first question is whether the truncation is happening on input or output. Input truncation means your retrieval or context assembly is sending too much. Fix this at the retrieval layer by tightening semantic similarity thresholds or chunking more aggressively. Output truncation means your tasks are producing longer outputs than the window allows—consider splitting tasks or using a model with a larger output limit.
Rising Truncation Rate Without Rising CUR
This is a pattern that catches teams off guard. It usually means a subset of requests are hitting the wall while the average CUR looks fine. Always look at CUR as a distribution, not just an average. The 95th percentile is what will hurt you.
Degrading PES Over Time
Prompt efficiency often erodes as teams add instructions incrementally—"just add a line to the system prompt" is the most common antipattern. Track PES on a rolling 14-day window. A downward trend signals prompt debt accumulating. Schedule a prompt audit before the degradation becomes customer-visible. For techniques to reverse this, Advanced Tokens and Context Windows: Going Beyond the Basics covers structured prompt compression and dynamic context assembly.
Benchmarking: What "Good" Looks Like
There are no universal benchmarks because task types vary too much, but here are working ranges for common workflow categories:
| Workflow Type | Typical CUR | Input/Output Ratio | CPT Range | |---|---|---|---| | Document summarization | 50–70% | 15:1–25:1 | $0.001–$0.005 | | Customer support drafting | 25–50% | 3:1–8:1 | $0.003–$0.015 | | Long-form content generation | 30–60% | 1:1–3:1 | $0.010–$0.05 | | Classification / routing | 10–25% | 10:1–50:1 | $0.0001–$0.001 | | RAG-based Q&A | 55–80% | 5:1–15:1 | $0.002–$0.020 |
Use these as orientation points, not hard targets. Your baseline is more useful than any industry average—establish it in week one and measure drift from there.
Model Selection and Context Window Economics
The decision of which model to use isn't just about capability—it's a context window and token-economics decision. Larger context windows command a price premium, and that premium is only worth paying when your CUR justifies it. As context window sizes continue expanding across the major providers, the calculus is shifting; Tokens and Context Windows: Trends and What to Expect in 2026 covers how pricing and capability trade-offs are likely to evolve.
The key discipline is to match model selection to measured CUR, not to theoretical maximum need. Run a smaller model first. Measure truncation rate. Upgrade only when the data demands it.
Building a Metrics Review Cadence
Metrics you don't review don't change behavior. Build a lightweight review cadence:
- Weekly: CPT and truncation rate by workflow. Flag anything more than 20% off baseline.
- Monthly: Full PES audit. Review CUR distributions for right-sizing opportunities. Assess whether model version updates have changed your token economics.
- Quarterly: Benchmark your stack against current model offerings. The market moves fast enough that a model that was the right choice six months ago may now be two generations behind.
This discipline is increasingly valuable as a professional skill. The ability to instrument, interpret, and optimize LLM workflows—not just use them—is what separates practitioners from power users. If you're building toward that kind of expertise, Tokens and Context Windows as a Career Skill frames why this matters professionally and how to demonstrate it.
Frequently Asked Questions
What's the easiest first metric to implement if we're starting from zero?
Start with Cost Per Task. It requires only the token counts from your API response and your model's current pricing, both of which are publicly available. CPT gives you immediate unit economics visibility and creates a baseline for every subsequent optimization effort.
Do input and output tokens need to be tracked separately?
Yes, always. Output tokens are priced at a meaningful premium on most APIs—often two to four times the input token rate. Tracking only total tokens obscures whether cost increases are driven by longer prompts, more verbose outputs, or both. The root cause determines the fix.
How does context window size affect quality, not just cost?
Models can exhibit attention degradation on very long contexts—the "lost in the middle" problem, where information in the middle of a long prompt receives less weight than information at the start or end. A CUR above 80–90% doesn't just risk truncation; it may produce lower-quality outputs even when nothing is cut off. This makes CUR a quality metric, not just a cost metric.
What's a reasonable truncation rate to tolerate in production?
Most production workflows should target below 0.5%. At 1–2%, you have a systemic problem affecting measurable user experience. Occasional truncation on edge cases (unusually long documents, unexpected user inputs) is acceptable if you have fallback handling; silent truncation with no fallback is never acceptable.
Should we measure tokens at the application level or the infrastructure level?
Both, if possible. Application-level measurement gives you workflow context and task-type segmentation. Infrastructure-level measurement (via your LLM gateway or reverse proxy) catches calls you didn't know were being made—third-party integrations, rogue experiments, misconfigured clients. The combination closes the gap.
How often should we recalibrate our baselines?
Recalibrate whenever you change models, update system prompts significantly, or introduce new task types. In a stable deployment, a monthly recalibration check is sufficient. After any significant change, establish a new baseline within the first week before you lose the ability to distinguish signal from noise.
Key Takeaways
- Six metrics cover the essentials: Cost Per Task, Context Utilization Rate, Input/Output Ratio, Prompt Efficiency Score, Truncation Rate, and Token Velocity.
- Log every API response in full, tagged by workflow, task type, model, and environment. Sampling hides tail behavior.
- Calculate CUR at log time and track it as a distribution—averages mask the 95th-percentile problems that cause real failures.
- Input and output tokens must be tracked separately because they carry different prices and point to different optimization levers.
- CUR below 30% signals over-provisioning; CUR above 85% signals truncation risk and potential quality degradation.
- Prompt debt is real and measurable: a declining PES trend over 14 days means your system prompts need an audit.
- Build a review cadence—weekly for operational metrics, monthly for efficiency audits, quarterly for model selection reassessment.
- Match model selection to measured CUR, not theoretical maximums. Right-size first; upgrade only when the data demands it.