There is no shortage of "it depends" advice about open versus closed source AI models. That advice is technically true and practically worthless. What teams actually need is a set of opinionated defaults, with the reasoning attached, so they can move fast and only deviate when they have a real reason.
These are the practices we would put in a playbook. Each comes with the why, because a practice you do not understand is one you will abandon the moment it is inconvenient. Adopt them as defaults, override them deliberately.
Default to Closed, Earn Your Way to Open
Start every new project on a closed API. The reason is leverage: a closed model gets you to a working product fastest, with frontier capability and zero infrastructure. Only migrate a workload to open-weight once it has earned that move by hitting real volume, a privacy requirement, or a customization need.
This is not a bias toward closed vendors. It is a bias toward proving value before paying infrastructure tax. Most prototypes die or pivot before they ever reach the volume where self-hosting pays off, so building infrastructure first is usually wasted effort.
What "Earning It" Looks Like
- Sustained, predictable volume where per-token API cost has become a real line item.
- A compliance rule that requires data to stay in your environment.
- A customization need an API cannot satisfy, like deep fine-tuning or distillation.
Abstract the Model Behind a Thin Interface
Never call a model provider's SDK directly throughout your codebase. Wrap it in a thin internal interface so that swapping models—open or closed—is a contained change in one place.
The payoff is enormous. Provider deprecations, pricing changes, and new model releases all become localized swaps rather than codebase-wide surgery. This single practice is what makes a routed portfolio strategy feasible at all, and it is cheap to do from day one.
Keep a Living Evaluation Set and Run It on Every Change
Your eval set is the most valuable asset in your AI stack. Build 30 to 100 representative examples with known good answers, and re-run it whenever you consider a new model, a new version, or a prompt change.
Without this, every model decision is a guess and every migration is a leap of faith. With it, you can answer "is this new model actually better for us?" in an hour. The teams that move fastest are the ones who never argue about model quality because they can just measure it. Our step-by-step approach covers how to build this set.
Measure Cost per Successful Task, Not Cost per Token
Per-token pricing is a trap when comparing models, because a cheaper model that fails more often costs more in retries, escalations, and human cleanup. Always normalize to cost per task that actually meets your quality bar.
Why This Changes Decisions
- A model at half the token price but a 70 percent success rate may be more expensive than a pricier model at 95 percent.
- It exposes false economies in self-hosting where a small open model looks cheap until you count its failure rate.
- It aligns the cost metric with the business outcome you actually care about.
Match the Privacy Mechanism to the Real Requirement
Do not over-engineer or under-engineer privacy. If your requirement is satisfied by a contractual no-retention guarantee, a closed enterprise tier is simpler and you should use it. If your requirement is that data physically never leaves your environment, only self-hosted open weights qualify, and no contract substitutes.
The mistake in both directions is expensive: self-hosting unnecessarily wastes engineering effort, while trusting a contractual promise where an architectural guarantee is required can fail an audit. Decide which one your situation truly needs. The common mistakes article details how teams get this wrong.
Run a Portfolio With Explicit Routing
The strongest production posture is rarely all-open or all-closed. Route each request to the cheapest model that can meet its quality bar: a closed frontier model for the hardest tasks, open-weight models for high-volume routine work.
This hedges vendor risk, controls cost, and keeps you near the frontier. It does require the thin interface and the eval set, which is why those practices come first. For a structured way to make routing decisions, see our framework article.
Revisit on a Schedule, Not on a Headline
Set a recurring calendar review—every three to six months—to re-run your eval set against current models and re-check pricing. Do not let a single dramatic headline trigger an impulsive migration, and do not let a year pass on autopilot either.
The cadence matters. Model capability and prices move fast enough that a stale decision quietly becomes a wrong one, but reactive thrashing on every announcement wastes engineering time. A scheduled review captures the upside of both discipline and freshness.
Treat Open and Closed as a Spectrum, Not a Binary
The framing "open versus closed" hides the most useful options, which live in the middle. Managed open-model hosting gives you open-weight economics and customization without owning raw GPUs. Closed enterprise tiers give you contractual privacy guarantees that satisfy many compliance needs without self-hosting. The best practice is to know the whole spectrum and pick the point that matches your requirement, not the nearest pole.
This matters because teams that think in binaries make worse decisions. They self-host on raw infrastructure when managed hosting would have served them, or they trust a basic API tier when an enterprise tier was the right fit. Mapping your requirement to the precise point on the spectrum—rather than to "open" or "closed"—consistently produces a better outcome at lower cost and risk.
The Spectrum in Order of Control
- Basic closed API: Least control, fastest start, cheapest at low volume.
- Closed enterprise tier: Contractual privacy and residency, still fully managed.
- Managed open hosting: Open-weight economics and customization, no infrastructure ownership.
- Self-hosted open weights: Maximum control and architectural privacy, full operational burden.
Make the Decision Reversible
A final, underrated practice: design every model decision to be cheap to reverse. The thin interface, the living eval set, and a portfolio mindset together mean that if a choice turns out wrong, you can back out in days rather than months. Irreversible decisions force you to over-deliberate up front; reversible ones let you decide quickly and correct fast.
This is why we favor shipping on a closed default and migrating as workloads earn it. That path is inherently reversible at each step, whereas committing to heavy self-hosted infrastructure early creates sunk costs that bias every later decision. Keep your options open, literally, and the pressure on any single decision drops.
Frequently Asked Questions
If closed is the default, when is open genuinely worth it?
When a workload has earned it: sustained high volume where API cost is significant, a hard data-residency requirement, or a customization need an API cannot meet. Until one of those is concrete, the infrastructure cost of open usually outweighs its benefits.
Is the thin interface worth it for a small project?
Yes, even more so. The cost is trivial up front and enormous to retrofit later. A small project is exactly the kind that will swap models several times as it finds its footing, and the abstraction makes each swap painless.
How do I keep my eval set from going stale?
Add new examples whenever you encounter a real failure in production. Treat each production miss as a test case to capture. Over time your eval set becomes a precise record of where models break for your specific use, which is far more valuable than any public benchmark.
Does running a portfolio add too much complexity?
It adds some, which is why the thin interface and eval set are prerequisites. Once those exist, routing is a manageable layer, and the cost and risk savings typically justify it for any team operating at meaningful scale.
Key Takeaways
- Default new projects to closed APIs; migrate workloads to open only once they earn it.
- Abstract every model call behind a thin interface so swaps stay contained.
- Maintain a living eval set and run it on every model or prompt change.
- Compare on cost per successful task, not cost per token, to avoid false economies.
- Match privacy to the real requirement and run a routed portfolio reviewed on a schedule.