You can manage AI cost with a calculator and discipline, but at any real scale you'll want tools. The right ones turn cost from a number that arrives once a month into a metric you watch in real time, attribute to features, and optimize deliberately. This article surveys the categories of tooling that matter, what each does, the trade-offs involved, and how to decide what your situation actually needs.
We're going to talk about categories rather than chase a list of specific products that will be out of date by next quarter. The categories are stable; the vendors within them churn. Understand what job each category does and you can evaluate any specific tool against your needs, today or a year from now.
A word of warning up front: tooling is a complement to understanding, not a substitute for it. A dashboard that shows you spending you don't understand is just a more expensive way to be confused. Read our Complete Guide first, then add tools to operationalize what you've learned.
Category 1: Token Counters and Estimators
The most basic tool is something that counts tokens before you send a request, so you can estimate cost and stay within context limits. Most providers ship a tokenizer library for exactly this, and they're free.
What they're for: Pre-flight cost estimation, enforcing input budgets, and verifying that your prompts are the size you think they are. They turn the manual arithmetic from our Step-by-Step Approach into a function call.
Trade-off: They tell you about individual requests, not aggregate spend. They're necessary but not sufficient — a starting point, not a cost-management strategy.
Category 2: Observability and Tracing Platforms
This is the category that matters most for teams running real workloads. LLM observability platforms log every request — input tokens, output tokens, model, latency, and crucially a feature tag — and present aggregate dashboards of where money goes.
What they're for: Per-feature cost attribution, spotting spikes, and answering the question "where is all this money going?" that the Case Study team had to answer under pressure. They make spend observable, which is the precondition for every optimization.
What to look for
- Per-feature and per-user attribution, not just a global total.
- Token-level logging, so you can see input versus output breakdowns.
- Alerting on budget thresholds.
- Cache hit visibility, so you can confirm caching is actually applying.
Trade-off: These add a dependency and sometimes a per-request overhead. For small projects they're overkill; for anything with real traffic they pay for themselves the first time they catch a runaway feature.
Category 3: Gateways and Routers
A gateway sits between your application and the model providers, giving you a single point to enforce policy. The most valuable cost feature is automatic model routing — sending simple requests to cheap models and hard ones to expensive models without that logic cluttering your application code.
What they're for: Centralizing model selection, enforcing spend caps, failing over between providers, and applying caching consistently. They operationalize the tiering discipline from our Framework at the infrastructure layer.
Trade-off: A gateway is another component in your critical path. It adds a small latency hop and an operational dependency. The benefit — centralized cost control and routing — usually justifies it once you're running multiple workloads, but a single simple feature may not need one.
Category 4: Provider-Native Cost Dashboards
Every major provider offers a built-in usage and billing dashboard. These are free, require no integration, and show your spend broken down by model and sometimes by API key.
What they're for: The baseline view of what you're spending, and setting hard spend caps. Use API keys per feature and the native dashboard gives you crude per-feature attribution for free.
Trade-off: They're provider-specific, so if you use multiple providers you get a fragmented picture. The attribution is coarse, and they're reactive — they show yesterday's spend, not a live alert. They're a floor, not a ceiling.
Category 5: Batch and Caching Infrastructure
Not a monitoring tool but an optimization one: the provider features that actually deliver savings. Batch APIs and prompt caching are built into the providers themselves, and the "tool" is knowing how to use them and confirming they're engaged.
What they're for: Directly cutting cost — batch at roughly half price, caching at a steep discount on repeated content. These aren't dashboards; they're the levers the dashboards help you decide to pull. Our Best Practices article ranks their impact.
Trade-off: They require code changes and verification. Caching configured but not hitting gives you cost without saving, which is why observability that surfaces cache hit rates pairs so well with them.
How to Choose Your Stack
Match tooling to your stage:
- Just experimenting: A token counter and the provider's native dashboard. Set a spend cap and move on.
- One live workload with real traffic: Add an observability platform for per-feature attribution and alerting. This is the highest-value addition.
- Multiple workloads or providers: Add a gateway for centralized routing, caching, and spend policy.
- Mature, high-volume system: All of the above, plus disciplined use of batch and caching infrastructure and a quarterly review.
Resist the urge to over-tool early. A small project drowning in dashboards has spent effort it should have spent shipping. Add each layer when its specific pain — invisible spend, scattered routing logic, fragmented billing — actually shows up.
A useful way to think about it: tools fall into two jobs, seeing and saving. Token counters, observability platforms, and native dashboards help you see — they make spend visible and attributable. Gateways, batch APIs, and caching infrastructure help you save — they actually move the bill. You generally want one good seeing tool before you invest heavily in saving tools, because without visibility you can't tell whether a saving tool is working. The observability platform that surfaces cache hit rates, for instance, is what tells you your caching is actually engaged rather than silently misconfigured. Pair a seeing tool with each saving tool and you close the loop: you make a change, you watch its effect, and you keep what works.
Frequently Asked Questions
What's the single most valuable tool for cost control?
An LLM observability platform with per-feature attribution. You cannot optimize what you cannot see, and the ability to know exactly which feature drives spend is what turns a scary monthly total into a targeted optimization plan. For any workload with real traffic, it's the highest-value addition.
Do I need a gateway, or can I route models in my own code?
For a single workload, in-code routing is fine and avoids an extra dependency. A gateway earns its place when you have multiple workloads or providers and want centralized routing, caching, spend caps, and failover in one place rather than duplicated across services. Add it when that duplication starts to hurt.
Are free provider dashboards enough?
For experimentation and small projects, often yes — set a spend cap and check the dashboard occasionally. They fall short once you need live alerting, fine-grained per-feature attribution, or a unified view across multiple providers. At that point a dedicated observability layer becomes worth its cost.
How do tools relate to the optimizations themselves?
Tools make optimizations visible and enforceable, but the savings come from the optimizations: smaller models, caching, trimming, batching. A dashboard that shows spend you don't act on saves nothing. Use tooling to decide what to optimize and to verify it worked, not as a substitute for the underlying discipline.
How do I avoid over-investing in tooling too early?
Add each layer only when its specific pain appears. Start with a free token counter and native dashboard. Add observability when spend goes invisible, a gateway when routing logic scatters across services, and advanced optimization infrastructure when volume justifies it. Tooling should follow need, not anticipate it.
Key Takeaways
- Token counters handle pre-flight estimation but don't track aggregate spend.
- Observability platforms with per-feature attribution are the highest-value cost tool for real workloads.
- Gateways centralize routing, caching, and spend policy across multiple workloads or providers.
- Provider-native dashboards are a free baseline but coarse and reactive.
- Batch APIs and prompt caching are the levers that actually deliver savings — verify they're engaged.
- Match your tooling stack to your stage and add each layer only when its pain appears.