A digital health startup hired an AI agency in Boston to build a patient triage chatbot in mid-2025. The chatbot collected symptoms, medical history, and insurance information to route patients to appropriate care pathways. The agency used a commercial LLM API to power the natural language understanding. Six months after launch, an HHS Office for Civil Rights (OCR) investigation revealed that patient data—including protected health information—was being sent to the LLM provider's API without a Business Associate Agreement in place. The LLM provider's terms of service allowed them to use API inputs for model improvement, meaning patient health data was potentially being incorporated into model training. The resulting HIPAA violation carried a $1.8 million fine for the healthcare startup and triggered a complete shutdown of the chatbot while the compliance failure was remediated. The AI agency was not fined directly but lost the client, two other healthcare prospects who learned about the incident, and faced an errors and omissions insurance claim.
Healthcare AI is one of the largest and fastest-growing markets for AI agencies. It is also one of the most dangerous from a compliance perspective. HIPAA has been law since 1996, and its privacy and security requirements are well-established. But the intersection of HIPAA with modern AI architectures creates novel compliance challenges that many agencies are not prepared for.
This post covers the HIPAA compliance framework as it applies to AI deployments, the specific technical and organizational requirements your agency must meet, and the governance structures that keep healthcare AI engagements compliant.
HIPAA Fundamentals for AI Agencies
Who HIPAA Applies To
Covered entities: Healthcare providers, health plans, and healthcare clearinghouses. Your healthcare clients are almost certainly covered entities.
Business associates: Any organization that creates, receives, maintains, or transmits protected health information (PHI) on behalf of a covered entity. If your agency handles PHI in any way during the course of building, deploying, or maintaining AI for a healthcare client, you are a business associate.
Business associate subcontractors: If your agency uses third-party services (cloud providers, LLM APIs, database services) that will receive PHI, those services are your subcontractors and must also comply with HIPAA requirements.
The chain of compliance: Your client (covered entity) must have a Business Associate Agreement (BAA) with your agency. Your agency must have BAAs with every subcontractor that handles PHI. Every link in this chain must be compliant, or the entire chain fails.
What PHI Is
Protected health information includes any individually identifiable health information that is created, received, maintained, or transmitted by a covered entity or business associate. This includes:
- Obvious PHI: Diagnoses, medications, lab results, treatment records
- Less obvious PHI: Appointment schedules, billing records, insurance information, patient demographics linked to health data
- AI-specific PHI concerns: Prompts containing patient information, embeddings generated from patient data, model outputs that reference patient health status, logs containing patient queries
The key test: If data can be linked to a specific individual and relates to their health, healthcare, or payment for healthcare, it is PHI and subject to HIPAA protections.
De-identification
HIPAA provides two methods for de-identifying PHI so that it is no longer subject to HIPAA restrictions:
Expert determination: A qualified statistical expert determines that the risk of identifying an individual from the data is very small.
Safe harbor: All 18 specified identifiers are removed, and there is no actual knowledge that the remaining information could identify an individual. The 18 identifiers include names, dates (except year), geographic data smaller than state, phone numbers, email addresses, Social Security numbers, medical record numbers, and others.
For AI applications, de-identification is often the preferred approach because it allows you to work with health data without the full HIPAA compliance burden. However, de-identification for AI is tricky—embeddings, model fine-tuning data, and prompt context can retain enough information to re-identify individuals even when traditional identifiers are removed.
AI-Specific HIPAA Challenges
LLM API Calls with PHI
The most common HIPAA violation in AI deployments is sending PHI to external LLM APIs without proper safeguards.
The problem: When your AI application sends a prompt to an LLM API that contains patient information (symptoms described in the patient's words, medical history, demographic details), that PHI is being transmitted to a third party. That third party must be covered by a BAA, and the transmission must meet HIPAA security requirements.
What you need:
- A BAA with the LLM provider that covers API usage
- Confirmation that the provider does not use API inputs for model training (or that the BAA permits such use, which is unlikely)
- Encryption of data in transit (TLS 1.2 or higher)
- Audit logging of all API calls containing PHI
- Data residency confirmation (PHI must be processed in approved jurisdictions)
Which providers offer healthcare-grade APIs:
As of 2026, major LLM providers including OpenAI, Anthropic, Google, and Microsoft offer enterprise API tiers with BAA support. However, the standard API tiers typically do not include BAA coverage. You must use the enterprise or healthcare-specific tiers and have the BAA executed before processing any PHI.
Embeddings and Vector Databases
When you generate embeddings from patient data and store them in vector databases, those embeddings may constitute PHI.
The compliance question: Can an embedding be used to re-identify a patient? Research has shown that embeddings can contain sufficient information to reconstruct aspects of the original text, potentially including identifying information. Until there is clear regulatory guidance otherwise, treat embeddings generated from PHI as PHI.
Implications:
- Vector databases storing embeddings from PHI must be HIPAA-compliant
- BAAs are needed with vector database providers
- Access controls, encryption, and audit logging apply to embedding storage
- Retention and deletion policies apply to embeddings
Fine-Tuning with Patient Data
Fine-tuning models on patient data creates additional compliance obligations.
- The fine-tuning platform must have a BAA in place
- Training data containing PHI must be encrypted at rest and in transit
- Access to fine-tuned models must be controlled (a model fine-tuned on PHI may memorize and reproduce patient information)
- Model outputs must be evaluated for potential PHI leakage
- When the engagement ends, fine-tuned models containing PHI must be handled according to the BAA's data disposition requirements
Logging and Monitoring
AI systems generate extensive logs—prompts, responses, error messages, performance metrics. If those logs contain PHI, they are subject to HIPAA requirements.
- Log storage must be encrypted and access-controlled
- Log retention must comply with HIPAA retention requirements (six years for certain records)
- Logs must be included in your client's right to access requests
- Log aggregation services and monitoring tools must be covered by BAAs
Technical Security Requirements
The HIPAA Security Rule
The HIPAA Security Rule requires administrative, physical, and technical safeguards for electronic PHI (ePHI).
Administrative safeguards:
- Designate a security official responsible for HIPAA compliance
- Conduct regular risk assessments of your AI systems and the PHI they handle
- Implement workforce training on HIPAA requirements
- Establish access management procedures
- Develop and test an incident response plan
Physical safeguards:
- Control physical access to systems that process or store PHI
- Implement workstation security policies
- Establish device and media controls for systems containing PHI
Technical safeguards:
- Access controls: Unique user identification, emergency access procedures, automatic logoff, encryption
- Audit controls: Record and examine activity in systems containing PHI
- Integrity controls: Protect ePHI from improper alteration or destruction
- Transmission security: Encrypt PHI in transit
Applying Security Requirements to AI Architecture
API layer: All API endpoints that receive or transmit PHI must use TLS 1.2+. Authenticate all API calls. Log all API calls with timestamps and caller identity.
Processing layer: The compute environment where AI models process PHI must be isolated and access-controlled. Use dedicated instances or verified multi-tenant environments with strong isolation guarantees.
Storage layer: All storage containing PHI (databases, vector stores, file systems, object storage) must be encrypted at rest with keys managed according to HIPAA requirements. Access must be logged.
Model layer: AI models that have been fine-tuned on PHI or that process PHI at inference time must be treated as PHI-containing assets. Access controls and audit logging apply.
Monitoring layer: Monitoring and observability tools that receive PHI-containing logs must be HIPAA-compliant.
Minimum Necessary Standard
HIPAA's minimum necessary standard requires that you limit PHI access and use to the minimum amount necessary to accomplish the intended purpose.
For AI systems, this means:
- Do not include PHI in prompts unless it is necessary for the AI task
- Strip unnecessary PHI from data before processing
- Use de-identification when the full PHI is not required
- Limit the scope of data accessed by AI models to what is needed for the specific use case
- Do not retain PHI longer than necessary
Governance Framework
Pre-Engagement Requirements
Before starting any healthcare AI engagement:
BAA execution: Execute a BAA with your client. Do not begin work, do not access any data, and do not start building until the BAA is signed.
Subcontractor BAA inventory: Identify every third-party service that will handle PHI and ensure BAAs are in place with each one.
Risk assessment: Conduct a risk assessment specific to the engagement. Identify the PHI that will be involved, the systems that will process it, the threats to that PHI, and the safeguards you will implement.
Data flow mapping: Document how PHI flows through your AI system—from collection through processing, storage, and output. Every point where PHI exists must have appropriate safeguards.
During Engagement
Access management: Grant PHI access only to team members who need it for their role. Use role-based access controls. Revoke access promptly when team members change roles or leave the project.
Incident detection and response: Monitor for security incidents. If a breach occurs, HIPAA requires notification to affected individuals within 60 days, notification to HHS, and (for breaches affecting 500+ individuals) notification to media. Have your incident response plan documented and practiced.
Change management: Any changes to the AI system that affect how PHI is processed must go through a change management process that includes security and compliance review.
Training: Ensure all team members working on the engagement complete HIPAA training before accessing PHI.
Post-Engagement
Data disposition: When the engagement ends, handle PHI according to the BAA's data disposition requirements. This typically means returning or destroying all PHI in your possession. For AI systems, this includes embeddings, fine-tuned models, logs, and any other data that contains or was derived from PHI.
Record retention: Retain documentation of your HIPAA compliance activities (risk assessments, training records, audit logs, incident reports) for six years after the engagement ends.
Lessons learned: Document what worked and what could be improved in your HIPAA compliance approach. Feed these lessons into your governance framework for future engagements.
The Business Associate Agreement
The BAA is the legal document that governs your relationship with your healthcare client regarding PHI. It is not a formality—it is a binding legal agreement with serious consequences for non-compliance.
Key BAA provisions your agency should understand:
- Permitted uses and disclosures: Defines exactly what you can do with PHI. Do not exceed these permissions.
- Safeguard requirements: Specifies the safeguards you must implement. These must match or exceed what you have committed to in your risk assessment.
- Breach notification: Defines your obligation to report security incidents to your client. Typically requires notification within a short timeframe (24-72 hours).
- Subcontractor requirements: Requires you to impose the same HIPAA obligations on your subcontractors.
- Data return and destruction: Defines what happens to PHI when the engagement ends.
- Termination provisions: Defines the consequences of HIPAA violations, up to and including agreement termination.
Negotiate your BAA carefully. Standard BAAs often impose obligations that are difficult for AI agencies to meet. Review the BAA with healthcare-experienced legal counsel before signing.
Common Healthcare AI Compliance Failures
Sending PHI to standard LLM APIs. The standard API tier of most LLM providers does not include BAA coverage or HIPAA-compliant data handling. Using these APIs with PHI is a violation.
Assuming de-identification is easy. De-identifying clinical text is much harder than removing structured identifiers. Patient narratives often contain information that can re-identify individuals (rare conditions, unique circumstances, detailed location references).
Ignoring the minimum necessary standard. Including full patient records in prompts when only a subset of information is needed violates the minimum necessary standard.
Not treating embeddings as PHI. Embeddings generated from PHI can contain re-identifiable information. Treat them with the same protections as the underlying PHI.
Skipping the risk assessment. A HIPAA risk assessment is not optional. It is a required administrative safeguard. Skipping it is itself a compliance violation.
Using personal devices to access PHI. If your team accesses PHI from personal laptops or phones without proper device management and encryption, you have a security gap.
Your Next Step
If healthcare AI is part of your agency's strategy, start by establishing your HIPAA compliance foundation. Get a HIPAA risk assessment template, identify a healthcare-experienced attorney to review your BAAs, and inventory the third-party services you use to determine which ones offer BAA-supported healthcare tiers.
Then build a healthcare AI engagement checklist: BAA executed, subcontractor BAAs confirmed, risk assessment completed, data flow documented, team trained, monitoring configured. Do not start building until every item on the checklist is complete.
The healthcare AI market rewards agencies that take compliance seriously. Healthcare clients have been burned by vendors who treated HIPAA as an afterthought, and they are looking for partners who demonstrate compliance maturity from the first conversation. Be that partner, and you will access one of the most lucrative and growing segments of the AI agency market.