A financial services firm hired an AI agency to help them select and implement a fraud detection platform. The agency recommended a vendor based primarily on impressive demo performance and competitive pricing. Six months into implementation, the firm discovered that the vendor's model had been trained predominantly on North American transaction patterns and performed poorly on their significant European and Asian transaction volumes. The vendor's data processing infrastructure was hosted in a jurisdiction that created GDPR compliance complications. And the vendor's contract included a clause allowing them to use the firm's transaction data to improve their models—something the firm's regulators would never approve. The implementation was abandoned. The firm had spent $890,000 on licensing, integration, and consulting fees. The AI agency lost the client and took a significant reputational hit. Every one of these issues was discoverable through proper due diligence.
AI vendor due diligence is not optional and it is not a formality. It is the structured process of verifying that an AI vendor can deliver what they promise, within the constraints that matter. For AI agencies, due diligence is a core competency. Your clients trust you to recommend vendors. When a recommendation fails, you bear the professional consequences even if the vendor bears the contractual ones.
Why AI Vendor Due Diligence Is More Complex Than Software Vendor Diligence
Standard software vendor due diligence covers financial stability, security posture, service levels, and contract terms. AI vendor due diligence covers all of that plus several AI-specific dimensions that create additional risk.
Model performance claims need independent verification. AI vendors present carefully curated benchmarks, often on datasets that favor their model's strengths. Real-world performance on your client's data may be dramatically different. Due diligence must include testing on representative data.
Data practices create unique risks. AI vendors may use client data to train or improve their models, share data with third parties, or process data in jurisdictions that create compliance problems. These practices are often buried in terms of service that nobody reads.
AI-specific regulatory requirements apply. Depending on the use case and jurisdiction, AI vendors may need to comply with AI-specific regulations (EU AI Act, state-level AI laws, sector-specific AI guidance). A vendor that is compliant for one use case may not be compliant for another.
Explainability and transparency vary widely. Some AI vendors provide detailed explanations of how their models work. Others treat their models as black boxes. The level of transparency required depends on the use case, the regulatory environment, and the client's governance requirements.
Vendor viability is harder to assess. The AI vendor landscape is volatile. Many AI companies are venture-funded startups burning through capital. A vendor that exists today may not exist in two years. Due diligence must assess long-term viability.
The AI Vendor Due Diligence Framework
Phase 1: Preliminary Screening
Before investing significant time in detailed due diligence, conduct a preliminary screening to eliminate obviously unsuitable vendors.
Basic qualification criteria:
- Does the vendor's solution address the specific use case?
- Does the vendor serve clients in the same industry and of similar size?
- Is the vendor's pricing within the client's budget range?
- Does the vendor operate in and support the relevant jurisdictions?
- Is the vendor willing to undergo your due diligence process?
Red flags that warrant immediate disqualification:
- Vendor refuses to provide references
- Vendor cannot explain how their model works at any level
- Vendor's terms of service claim ownership of client data
- Vendor has no SOC 2 or equivalent security certification
- Vendor has active lawsuits related to their AI technology
- Vendor cannot provide a Data Processing Agreement
Phase 2: Technical Due Diligence
Technical due diligence evaluates whether the vendor's solution actually works as claimed and whether it can be integrated into the client's technical environment.
Model performance evaluation:
- Request a proof of concept or pilot on representative data. Do not rely solely on vendor-provided benchmarks
- Define your own evaluation metrics based on the specific use case requirements
- Test performance across different data segments, including edge cases and minority groups
- Evaluate performance over time to assess stability and consistency
- Compare vendor performance to baseline alternatives (simpler models, rules-based systems, competitor solutions)
Architecture and integration assessment:
- How does the vendor's solution integrate with the client's existing systems?
- What APIs are available and how mature are they?
- What is the expected latency and throughput?
- Can the solution scale to handle the client's data volumes?
- What are the infrastructure requirements (cloud provider, compute, storage)?
- Is the solution available as SaaS, on-premises, or both?
Data requirements and handling:
- What data does the solution need to function?
- How is data ingested (batch, streaming, API)?
- Where is data processed and stored?
- What data formats are supported?
- How is data encrypted at rest and in transit?
- What happens to client data when the contract ends?
Model management capabilities:
- How are models updated? How frequently?
- Can clients control when updates are applied?
- Is there a rollback capability?
- How is model drift monitored?
- Can clients fine-tune or customize models?
- What monitoring and alerting capabilities are available?
Reliability and disaster recovery:
- What are the uptime guarantees (SLA)?
- What is the disaster recovery plan?
- What are the recovery time and recovery point objectives?
- Has the vendor experienced significant outages? How were they handled?
- Is there geographic redundancy?
Phase 3: Security Due Diligence
Security due diligence for AI vendors includes traditional information security assessment plus AI-specific security considerations.
Information security standards:
- Does the vendor hold SOC 2 Type II certification? Request the report
- Is the vendor ISO 27001 certified?
- Does the vendor have a dedicated security team?
- What is the vendor's vulnerability management process?
- Has the vendor undergone a recent penetration test? What were the findings?
- Does the vendor have a security incident response plan?
AI-specific security:
- How is the vendor protecting against adversarial attacks on their models?
- What safeguards prevent prompt injection or data poisoning?
- How is model access controlled?
- Are model weights and training data protected?
- Has the vendor assessed their AI-specific attack surface?
Data security:
- Who within the vendor organization can access client data?
- Are there background checks for personnel with data access?
- How is access logged and monitored?
- Is there a data breach notification process?
- Does the vendor sub-process data to any third parties?
Phase 4: Compliance and Regulatory Due Diligence
This phase assesses whether the vendor's solution can operate within the client's regulatory environment.
General compliance:
- GDPR compliance (for European data subjects)
- CCPA/CPRA compliance (for California residents)
- Sector-specific regulations (HIPAA, GLBA, PCI-DSS, as applicable)
- Data localization requirements (where data must be processed and stored)
AI-specific compliance:
- EU AI Act compliance (risk classification, transparency requirements, conformity assessments)
- State-level AI regulations (bias auditing, disclosure requirements)
- Sector-specific AI guidance (OCC, Fed, EEOC, HUD, as applicable)
- International AI regulations (Canada's AIDA, UK's pro-innovation framework, as applicable)
Compliance documentation:
- Can the vendor provide a Data Processing Agreement (DPA)?
- Can the vendor demonstrate compliance through certifications or audit reports?
- Does the vendor have a compliance team or designated compliance officer?
- How does the vendor track and respond to regulatory changes?
Phase 5: Ethical and Responsible AI Due Diligence
This phase evaluates the vendor's commitment to responsible AI practices.
Bias and fairness:
- Has the vendor tested their models for bias across relevant protected categories?
- Are bias testing results documented and available?
- What bias mitigation measures are in place?
- Does the vendor commit to ongoing bias monitoring?
Transparency and explainability:
- Can the vendor explain how their model reaches its outputs?
- What level of explainability is available (global model explanation, individual prediction explanation)?
- Can the vendor provide sufficient transparency for regulatory requirements?
- Are there limitations on explainability that the client should be aware of?
Human oversight:
- Does the solution support human-in-the-loop decision-making where required?
- Can human operators override model outputs?
- Are there configurable confidence thresholds for automated versus human-reviewed decisions?
Training data ethics:
- Was the training data collected with appropriate consent?
- Were data subjects compensated or credited where appropriate?
- Are there known ethical concerns about the training data?
Phase 6: Commercial and Financial Due Diligence
Assess the vendor's financial health and the commercial terms of the engagement.
Financial stability:
- How is the vendor funded? What is their cash position?
- Are they profitable or what is their path to profitability?
- Who are their investors?
- What is their revenue trajectory?
- Are there any pending legal actions that could affect financial stability?
Contract terms analysis:
- Data ownership: Does the client retain ownership of their data? This is non-negotiable
- Data usage: Can the vendor use client data to train or improve their models? This must be explicitly prohibited unless the client consents
- IP rights: Who owns any models, derivatives, or outputs created using the vendor's solution with client data?
- Liability and indemnification: What is the vendor's liability for errors, bias, or compliance failures?
- Exit terms: What happens when the contract ends? How is data returned or destroyed? What is the transition period?
- SLA and remedies: What service levels are guaranteed and what are the penalties for failure?
- Pricing stability: How can pricing change over the contract term? Are there caps on increases?
Vendor lock-in assessment:
- How difficult would it be to switch to an alternative vendor?
- Is client data portable? In what format?
- Are there proprietary integrations that would need to be rebuilt?
- Does the vendor use proprietary data formats or APIs that increase switching costs?
Phase 7: Reference Checks
Never skip reference checks. They provide information you cannot get from the vendor directly.
Reference call structure:
- Ask for references in the same industry and of similar size to the client
- Request references for the same use case, not just general references
- Ask about implementation experience, support quality, and actual performance versus promised performance
- Ask about what went wrong and how the vendor handled it
- Ask if they would choose the same vendor again
Questions to ask references:
- How long have you been using the vendor's solution?
- Did the solution perform as expected based on the sales process?
- What was the implementation experience like? Did it take longer or cost more than planned?
- How responsive is the vendor's support team?
- Have you experienced any data or security issues?
- What would you change about the vendor or their solution?
- Would you recommend this vendor to a peer?
Documenting and Communicating Due Diligence Findings
Due diligence findings should be documented in a structured report that enables decision-making.
The Due Diligence Report should include:
- Executive summary with overall recommendation (proceed, proceed with conditions, do not proceed)
- Findings by phase with risk ratings (low, medium, high, critical)
- Identified risks and proposed mitigations
- Comparison to alternative vendors (if multiple vendors were evaluated)
- Conditions that must be met before proceeding (contract changes, technical requirements, compliance actions)
- Ongoing monitoring requirements if the vendor is selected
Present findings objectively. Your role as an AI agency is to provide the client with the information they need to make an informed decision. Present risks honestly, even if they complicate the recommendation. Clients appreciate candor more than they appreciate cheerleading.
Building Due Diligence Into Your Agency's Process
Create standardized templates. Build templates for each phase of due diligence so that the process is consistent across projects and team members.
Allocate sufficient time and budget. Due diligence takes time. For a Tier 3 or Tier 4 vendor (high risk, high spend), expect four to eight weeks of diligence work. Build this time into project timelines and proposals.
Maintain a vendor knowledge base. Once you have completed due diligence on a vendor, maintain the findings in a central knowledge base. If another client is considering the same vendor, you have a head start. Update findings as the vendor evolves.
Stay current on the landscape. The AI vendor market moves fast. Continuously track vendor developments, new entrants, and industry trends so that your due diligence is informed by current knowledge.
Your Next Step
Select the next AI vendor your agency plans to recommend to a client. Before making the recommendation, run them through this due diligence framework—even if informally. Focus on the three areas where surprises are most common and most expensive: data practices (how they handle client data), compliance posture (whether they meet regulatory requirements), and contract terms (especially data ownership and exit provisions). Document what you find and share it with your client before they sign anything.