When people list the risks of foundation models, they say "hallucination" and stop. Hallucination is real, but it is the obvious risk, the one everyone already guards against. The risks that quietly damage organizations are the ones nobody puts on a slide: the model that silently changes behavior when the provider updates it, the sensitive data that leaks through prompts, the vendor dependency you discover only when pricing or availability changes, and the ungoverned usage spreading through teams faster than any policy.
This article surfaces the non-obvious risks and pairs each with a concrete mitigation. The framing is deliberately unglamorous, because these are the failures that produce real incidents and the ones most teams have no plan for. If your risk register for AI says "the model might be wrong," you are managing the easy 20% and ignoring the hard 80%.
Silent Model Drift
The risk almost nobody plans for: the model you built on changes underneath you. Providers update hosted models, sometimes without a version bump you notice, and behavior shifts. Prompts that worked stop working. Output format changes subtly. A workflow that was reliable for months degrades and nobody knows why because nothing in your code changed.
Why it is dangerous
Drift is insidious because there is no error, no alert, no failed deploy. The system keeps running and quietly gets worse. By the time someone notices the quality drop, you have shipped degraded output for weeks.
How to manage it
- Pin versions where you can. Use explicitly versioned model identifiers rather than "latest" aliases so updates are a decision, not a surprise.
- Run a regression eval on a schedule. A small, fixed test set run daily or weekly catches behavior changes the moment they appear. This is the single most valuable safeguard against drift.
- Monitor output distribution in production. Track simple signals — output length, format validity, refusal rate — and alert on shifts. The mechanics of building these evals are covered in A Step-by-Step Approach to Foundation Models.
Data Leakage Through Prompts
The second hidden risk is that your sensitive data ends up somewhere it should not. People paste customer records, internal documents, and credentials into prompts because it is convenient. Depending on the provider and the contract, that data may be logged, retained, or used in ways your security team never approved.
The mitigation is partly technical and partly behavioral:
- Know your provider's data terms. Whether prompts are retained or used for training varies by provider and tier. This is a contract question, not a guess.
- Strip or tokenize sensitive fields before they reach the model when the task does not require them. The model rarely needs the actual account number to do its job.
- Make the safe path the easy path. People leak data because the convenient route runs through the prompt. Give them an approved workflow that handles sensitive cases. The behavioral side of this is covered in Rolling Out Foundation Models Across a Team.
Supply-Chain and Vendor Concentration
If your product depends on a single model from a single provider, you have a concentration risk most teams underestimate. The provider can change pricing, deprecate the model, alter rate limits, restrict use cases, or have an outage at the worst possible moment. You inherit all of those decisions.
Concrete mitigations
- Abstract the provider behind your own interface so swapping models is a configuration change, not a rewrite. This is cheap to do early and expensive to retrofit.
- Keep a tested fallback. A second provider or a smaller self-hostable model you can route to during an outage or a pricing shock. You do not need it running, but you need to have proven it works.
- Track unit economics continuously. A pricing change that doubles your per-call cost can turn a profitable feature into a loss overnight. Know your cost per transaction and your tolerance.
The selection logic for which models to depend on is in A Framework for Foundation Models.
Shadow AI and Ungoverned Usage
While leadership debates an AI policy, the team is already using foundation models for real work through personal accounts, browser extensions, and unapproved tools. This shadow usage carries every risk above with none of the governance. Data leaks through tools nobody vetted, quality varies wildly, and there is no record of what the model was used for.
The wrong response is a blanket ban, which drives usage further underground. The right response:
- Provide a sanctioned path that is genuinely good. People use shadow tools because the approved option is missing or worse. Beat the shadow tools on convenience.
- Inventory what is actually being used. You cannot govern what you cannot see. A blunt survey often reveals more than monitoring.
- Govern by use case, not by tool. Define what is acceptable to do, and the tool question becomes manageable.
Over-Reliance and Skill Atrophy
A subtler organizational risk: as people lean on the model, their own judgment can erode. A team that lets the model draft everything may lose the ability to evaluate whether a draft is good. This is not an argument against using the model — it is an argument for keeping humans in the loop on the judgment that matters, not just the typing.
The mitigation is structural. Require human review on high-stakes output, and make that review substantive rather than a rubber stamp. Rotate people through tasks without AI assistance occasionally so the underlying skill stays sharp. The teams that handle this well treat the model as an accelerant for skilled people, not a replacement for skill. The common over-reliance patterns are in 7 Common Mistakes with Foundation Models (and How to Avoid Them).
Bias and Inconsistent Treatment
Foundation models reflect patterns in their training data, including biases. The risk is not just offensive output — it is inconsistent treatment that surfaces in subtle, legally and ethically consequential ways: a hiring screen that ranks equivalent candidates differently based on name, a support system that responds differently to different dialects. These failures are hard to spot because they are statistical, not obvious in any single case.
Manage it by evaluating across demographic and input slices, not just on aggregate accuracy. If your eval shows 95% accuracy overall but a specific group sees materially worse results, you have a problem that the average is hiding. This slice-based evaluation is the only reliable way to catch bias before it becomes an incident.
Frequently Asked Questions
What is the most overlooked risk?
Silent model drift. Most teams have no regression eval, so when a hosted model's behavior changes, they ship degraded output for weeks before noticing. A small scheduled eval is the cheapest high-value safeguard you can add.
How do I prevent sensitive data leakage?
Understand your provider's data-retention terms, strip or tokenize sensitive fields the task does not need, and make the approved workflow more convenient than pasting raw data into a prompt. Behavior follows convenience more than policy.
Is depending on one provider really that risky?
Yes, for anything in production. Pricing changes, deprecations, rate limits, and outages are all decisions the provider makes that you inherit. Abstract the provider behind your own interface and keep a tested fallback.
How do we handle shadow AI usage?
Not with a ban, which drives it underground. Provide a sanctioned path that beats the shadow tools on convenience, inventory what is actually in use, and govern by use case rather than by tool.
How do we catch bias before it causes harm?
Evaluate across input and demographic slices instead of relying on aggregate accuracy. Bias hides in the average; it shows up when you look at the worst-performing group. Make slice-based evaluation a standard part of testing.
Key Takeaways
- Hallucination is the obvious risk; drift, leakage, vendor concentration, and shadow usage are the ones that sink projects.
- Guard against silent drift with pinned versions and a scheduled regression eval.
- Prevent data leakage by knowing provider terms, minimizing sensitive fields, and making the safe path convenient.
- Reduce vendor risk by abstracting the provider and keeping a tested fallback.
- Beat shadow AI with a genuinely good sanctioned option, and catch bias with slice-based evaluation.