A computer vision AI agency used a third-party image annotation service to label training data for a retail client's product recognition system. The annotation service employed workers in three countries to label product images. When the client's security team audited the AI agency's supply chain, they discovered that the annotation service had been storing copies of the product images—which included proprietary packaging designs for unreleased products—on servers in a jurisdiction with weak data protection laws. The workers had unrestricted access to the full image dataset. The annotation service's privacy policy explicitly permitted use of client data for service improvement, which could include training their own models. The retail client terminated the contract. The agency lost 420,000 dollars in annual revenue and spent months rebuilding its vendor management practices.
AI agencies depend on a complex supply chain of vendors, tools, and services. Cloud infrastructure, pre-trained models, data annotation services, monitoring platforms, development tools, and API-based AI services all introduce risks that your agency must manage. Vendor management governance ensures these dependencies do not become liabilities.
The AI Vendor Landscape
Types of AI Vendors
Cloud infrastructure providers. AWS, Azure, Google Cloud, and other providers host your development and production environments. They process and store your data and your clients' data.
Model providers. Companies that provide pre-trained models, foundation models, or model APIs. This includes OpenAI, Anthropic, Google, Meta, and many specialized model providers.
Data providers. Companies that provide training data, enrichment data, or real-time data feeds. This includes data marketplaces, web scraping services, and domain-specific data providers.
Annotation and labeling services. Companies that label training data, including both platform-based services and managed annotation teams.
Development tools. ML platforms, experiment tracking tools, feature stores, and other development infrastructure.
Monitoring and observability. Platforms for model monitoring, performance tracking, and alerting.
Specialized AI services. OCR, speech-to-text, NLP, computer vision, and other AI capabilities consumed as services.
Vendor Risk Categories
Data security risk. The vendor handles your data or your clients' data. A breach at the vendor exposes your data.
Data usage risk. The vendor may use your data for their own purposes—training their models, improving their services, or sharing with third parties.
Availability risk. Your systems depend on the vendor's availability. A vendor outage causes your systems to fail.
Quality risk. The vendor's outputs (model predictions, data quality, annotations) affect the quality of your systems. Quality degradation at the vendor cascades to your systems.
Compliance risk. The vendor's practices may not comply with regulations that apply to your systems. Their non-compliance becomes your non-compliance.
Concentration risk. Over-reliance on a single vendor creates a single point of failure that affects multiple projects.
Continuity risk. The vendor may change their terms, pricing, or capabilities, or may go out of business entirely.
Vendor Assessment Framework
Pre-Engagement Assessment
Before engaging any AI vendor, conduct a structured assessment:
Security assessment. Evaluate the vendor's security posture including security certifications (ISO 27001, SOC 2), data encryption practices (at rest and in transit), access control mechanisms, vulnerability management program, incident response capabilities, and employee security training.
Privacy assessment. Evaluate the vendor's privacy practices including data processing and retention policies, data usage rights (can they use your data?), data subject rights support, cross-border data transfer practices, and privacy certifications and compliance.
Compliance assessment. Evaluate the vendor's compliance with regulations relevant to your use case including GDPR compliance, HIPAA compliance (for healthcare data), PCI DSS compliance (for payment data), and AI-specific regulations (EU AI Act).
Operational assessment. Evaluate the vendor's operational capabilities including service level agreements and uptime history, support responsiveness and quality, change management practices, disaster recovery and business continuity, and scalability and performance.
Financial assessment. Evaluate the vendor's financial stability including financial health and funding status, customer base and revenue stability, market position and competitive dynamics, and pricing model and total cost of ownership.
Risk Scoring
Score each vendor on each risk category using a standardized scale. Combine scores into an overall vendor risk rating:
- Low risk. Vendor meets all requirements with no significant concerns. Standard monitoring is sufficient.
- Medium risk. Vendor meets most requirements with some concerns that are manageable with additional controls. Enhanced monitoring is required.
- High risk. Vendor has significant gaps or concerns. Engagement requires risk mitigation measures and close monitoring.
- Unacceptable. Vendor does not meet minimum requirements. Do not engage.
Contractual Governance
Essential Contract Provisions for AI Vendors
Data ownership and usage. Clearly specify that your data (and your clients' data) remains your property. The vendor may not use it for any purpose other than providing the contracted service. This is critical for model providers—many default terms of service grant the provider rights to use input data for model training.
Data protection. Require the vendor to implement specific security controls, comply with applicable data protection regulations, notify you promptly of any data breach, and cooperate with your security audits.
Confidentiality. Require the vendor to keep all information about your systems, your clients, and your engagements confidential. Include provisions that survive contract termination.
Service levels. Define specific service level agreements for availability, performance, support response time, and quality metrics. Include remedies for SLA breaches.
Change notification. Require the vendor to notify you in advance of significant changes to their service, including model updates, API changes, data handling changes, and security changes.
Audit rights. Reserve the right to audit the vendor's security, privacy, and compliance practices. Alternatively, require the vendor to provide independent audit reports (SOC 2, ISO 27001).
Subcontractor controls. Require the vendor to notify you of and obtain approval for subcontractors who will have access to your data. Require subcontractors to meet the same security and privacy standards.
Data return and deletion. Upon contract termination, the vendor must return all your data and certify its deletion from their systems.
Liability and indemnification. Define the vendor's liability for breaches, service failures, and non-compliance. Include indemnification provisions for losses resulting from the vendor's negligence.
Termination provisions. Include termination rights for cause (including material security or privacy breaches) and convenience (with appropriate notice periods). Define transition support obligations.
Model Provider-Specific Provisions
When contracting with model providers (foundation model APIs, pre-trained model services), include additional provisions:
Model behavior guarantees. Define expectations for model accuracy, consistency, and safety. Include provisions for the vendor to notify you of model updates that could affect behavior.
Input/output data handling. Explicitly prohibit the vendor from using your inputs or outputs for model training, improvement, or any purpose other than providing the service. Require data deletion after processing.
Content filtering. Understand and document the vendor's content filtering policies. Ensure they are compatible with your use case.
Rate limiting and availability. Define rate limits, queue management, and availability guarantees specific to your usage patterns.
Ongoing Vendor Monitoring
Periodic Reassessment
Reassess vendors on a regular schedule based on their risk level:
- High-risk vendors: Quarterly reassessment
- Medium-risk vendors: Semi-annual reassessment
- Low-risk vendors: Annual reassessment
Reassessment should include reviewing the vendor's security certifications and audit reports, assessing any changes to the vendor's security, privacy, or compliance posture, evaluating the vendor's performance against SLAs, reviewing any incidents or concerns that arose during the period, and updating the vendor's risk rating based on the reassessment.
Continuous Monitoring
Between formal reassessments, monitor vendors continuously:
- Track vendor uptime and performance metrics
- Monitor for vendor security incidents and breaches in the news
- Track vendor financial health indicators
- Monitor for changes to vendor terms of service and privacy policies
- Track vendor product changes and deprecations
Incident Management
When a vendor experiences a security incident, data breach, or service disruption that affects your agency:
Assess impact. Determine which of your systems and clients are affected. Assess the severity of the impact.
Contain. If possible, switch to alternative vendors or manual processes to contain the impact.
Communicate. Notify affected clients and internal stakeholders. Be transparent about the vendor nature of the incident.
Coordinate. Work with the vendor to understand the incident scope, obtain forensic information, and ensure remediation.
Document. Document the incident, your response, and the lessons learned. Update your vendor risk assessment.
Building a Vendor Governance Program
Vendor Inventory
Maintain a comprehensive inventory of all AI vendors including vendor name and contact information, services provided, data accessed or processed, contracts and agreements, risk classification, last assessment date, and responsible internal contact.
Approved Vendor List
Maintain an approved vendor list for commonly used services. Pre-assess vendors for security, privacy, and compliance. When team members need a service, they should choose from the approved list rather than engaging new vendors independently.
Vendor Onboarding Process
Define a standard onboarding process for new vendors that includes business justification and approval, security and privacy assessment, contract negotiation with required provisions, integration security review, and risk classification and monitoring plan.
Vendor Offboarding Process
Define a standard offboarding process that includes data return verification, data deletion certification, access revocation, credential rotation, and documentation archival.
Managing Vendor Concentration Risk
Identifying Concentration
Map your vendor dependencies across all projects. Identify vendors that are used across multiple clients or multiple critical systems. These represent concentration risk—a single vendor failure could affect multiple engagements simultaneously.
Create a vendor dependency map that shows which projects depend on which vendors, what data each vendor handles, what the impact would be if the vendor became unavailable, and how long it would take to switch to an alternative.
Mitigation Strategies
Multi-vendor strategy. For critical capabilities, maintain relationships with at least two vendors. Design your architecture to allow switching between vendors. This does not mean using two vendors simultaneously—it means having a tested alternative ready to deploy.
Abstraction layers. Build abstraction layers that decouple your systems from specific vendor implementations. For model APIs, build a provider abstraction that can switch between providers with configuration changes. For cloud services, use infrastructure-as-code that can deploy to multiple clouds. This reduces switching costs and enables flexibility.
Fallback plans. For each critical vendor dependency, define a fallback plan that specifies what you will do if the vendor becomes unavailable. Test fallback plans at least annually. A fallback plan that has never been tested is not a plan—it is a hope.
Contractual protections. Include contractual provisions that protect against vendor lock-in, such as data portability requirements, reasonable transition periods, prohibition on proprietary formats, and data export capabilities.
Vendor Governance Metrics
Track these metrics to assess the health of your vendor management program:
Vendor assessment completion rate. Percentage of vendors that have completed a security and privacy assessment within the required timeframe. Target: 100 percent.
Contract compliance rate. Percentage of vendor contracts that include all required provisions. Target: 100 percent.
Vendor incident rate. Number of vendor-related incidents per quarter, including security incidents, outages, and quality issues. Track trends over time.
Vendor reassessment timeliness. Percentage of vendor reassessments completed on schedule. Target: 100 percent.
Concentration risk score. A composite score that reflects the degree of vendor concentration in your portfolio. Monitor for increasing concentration and take action to diversify.
Vendor performance against SLAs. Track each vendor's performance against contractual service level agreements. Use this data for vendor management conversations and renewal decisions.
Emerging Vendor Risks in AI
AI Model Provider Risks
The rapid growth of foundation model providers creates new vendor risks:
Model deprecation. Providers may deprecate or change models that your systems depend on, breaking functionality or altering behavior.
Terms of service changes. Providers may change their terms of service to expand their rights to use your data, restrict certain use cases, or change pricing.
Quality inconsistency. Model quality can vary between versions. An update that improves general performance may degrade performance for your specific use case.
Availability and rate limiting. Model APIs may experience capacity constraints, rate limiting, or outages that affect your production systems.
Mitigate these risks by monitoring provider announcements, testing model updates before deploying them, maintaining fallback capabilities, and ensuring your architecture can accommodate provider changes.
Your Next Step
This week: Create or update your AI vendor inventory. List every vendor that handles your data, your clients' data, or provides capabilities that your AI systems depend on. For each vendor, note the data they access and the criticality of their service.
This month: Conduct security and privacy assessments for your highest-risk vendors. Review your contracts with these vendors to ensure they include the essential provisions outlined in this guide. Identify and address the most critical gaps.
This quarter: Build a formal vendor governance program with standard assessment templates, contractual requirements, and monitoring procedures. Establish an approved vendor list. Implement a vendor onboarding and offboarding process. Address vendor concentration risks identified in your inventory.