A 15-person AI agency in San Francisco built a customer churn prediction model for a SaaS company with 200,000 users. The model ingested usage data, billing history, support ticket text, and demographic information. It worked well—predicted churn with 87 percent accuracy. Then California's Attorney General opened an investigation into the SaaS company's data practices. The investigation uncovered that the agency had used personal information in ways not disclosed in the client's privacy policy. The agency had scraped publicly available social media profiles and matched them to customer records to enrich the training dataset. No one had checked whether this practice was disclosed or whether consumers had opted out. The resulting settlement cost the SaaS company 1.2 million dollars. The agency lost the client and spent four months rebuilding its data handling practices from scratch.
The California Consumer Privacy Act, as amended by the California Privacy Rights Act (CPRA), creates specific obligations that affect how AI agencies collect, use, and process personal information. If your agency handles data from California residents—and if you serve any meaningful US client base, you almost certainly do—CCPA compliance is not optional.
CCPA Fundamentals for AI Agencies
Who the Law Covers
The CCPA applies to for-profit businesses that do business in California and meet any of three thresholds: annual gross revenues exceeding 25 million dollars, annually buying, selling, or sharing the personal information of 100,000 or more California residents or households, or deriving 50 percent or more of annual revenue from selling or sharing California residents' personal information.
Your agency may be directly covered if it meets these thresholds. More commonly, AI agencies encounter CCPA obligations through their role as service providers processing personal information on behalf of covered clients. Under CCPA, a service provider processes personal information on behalf of a business pursuant to a written contract that specifies the business purpose for which the information is provided and prohibits the service provider from using it for other purposes.
What Counts as Personal Information
The CCPA defines personal information broadly. It includes information that identifies, relates to, describes, is reasonably capable of being associated with, or could reasonably be linked, directly or indirectly, with a particular consumer or household. For AI agencies, this encompasses far more than names and email addresses.
Data commonly used in AI that qualifies as personal information under CCPA:
- Names, email addresses, phone numbers, postal addresses
- Online identifiers including IP addresses, cookies, device IDs
- Browsing and search history
- Purchase history and consumer interaction records
- Geolocation data
- Professional or employment-related information
- Education information
- Inferences drawn from any of the above to create a consumer profile
That last category is critical for AI agencies. Inferences you generate using AI models—predictions about consumer behavior, preferences, characteristics, or tendencies—are themselves personal information under CCPA. This means the outputs of your models, not just the inputs, are regulated.
Sensitive Personal Information
The CPRA amendments created a category of sensitive personal information that carries additional restrictions. This includes Social Security numbers, financial account credentials, precise geolocation, racial or ethnic origin, religious beliefs, union membership, contents of mail or email or text messages, genetic data, biometric information, health information, and information about sex life or sexual orientation.
If your AI systems process sensitive personal information, consumers have the right to limit its use to what is necessary to perform the services they requested. This right to limit can significantly constrain what you can do with sensitive data in your models.
CCPA Rights That Affect AI Development
Right to Know
Consumers have the right to know what personal information a business collects, the categories of sources from which it was collected, the business or commercial purpose for collecting or selling it, and the categories of third parties with whom it is shared. For AI agencies, this means you must be able to explain what data your models use, where it came from, and why you are using it.
Right to Delete
Consumers can request deletion of their personal information. For AI agencies, this raises complex questions. If a consumer whose data was used to train a model requests deletion, do you need to retrain the model? The current regulatory guidance suggests that if the data has been aggregated, de-identified, or used only in training and cannot be extracted from the model, deletion of the source data may be sufficient. However, this area is evolving and you should consult legal counsel for your specific circumstances.
Right to Opt Out of Sale and Sharing
Consumers can opt out of the sale or sharing of their personal information. Sharing includes making personal information available to third parties for cross-context behavioral advertising, which is relevant if your AI systems support advertising or personalization use cases.
Right to Correct
Consumers can request correction of inaccurate personal information. For AI agencies, this could mean correcting data used as model inputs or correcting inferences generated by models.
Right to Limit Use of Sensitive Personal Information
As noted above, consumers can limit the use of sensitive personal information to what is necessary for the requested service. This right can restrict your ability to use sensitive data in model training or feature engineering.
Right to Non-Discrimination
Businesses cannot discriminate against consumers who exercise their CCPA rights. This means you cannot provide a worse service, charge a different price, or deny a service because a consumer opted out or requested deletion.
The Service Provider Framework
Most AI agencies operate as service providers under CCPA. This framework creates both protections and obligations.
Service Provider Agreements
Your contracts with covered clients must include specific CCPA-compliant provisions:
- Specified business purpose. The contract must specify the business purpose for which personal information is disclosed to your agency.
- Use restriction. You must agree not to retain, use, or disclose personal information for any purpose other than the specified business purpose, including for a commercial purpose other than providing the services.
- No selling or sharing. You must agree not to sell or share personal information you receive from the client.
- Compliance assistance. You must agree to assist the client in responding to consumer rights requests.
- Sub-processor obligations. If you engage sub-processors (including cloud providers, third-party AI tools, or data enrichment services), they must be bound by equivalent obligations.
- Notification and remediation. You must notify the client if you determine you can no longer meet your CCPA obligations and allow the client to take reasonable steps to stop and remediate unauthorized use.
What Service Providers Can and Cannot Do
You can use personal information to perform the services specified in your contract, to detect security incidents, to debug and repair errors, and for certain internal uses that are compatible with the business purpose and consumer expectations.
You cannot use personal information for purposes other than the specified business purpose, sell or share the information, combine it with personal information received from other sources (with limited exceptions), or retain it beyond what is necessary for the specified purpose.
For AI agencies, these restrictions have practical implications. You generally cannot use one client's data to train models for another client. You cannot build a proprietary dataset by aggregating personal information from multiple client engagements. You cannot use personal information you receive as a service provider for your own commercial purposes.
Data Mapping for AI Compliance
Before you can comply with CCPA, you need to understand what data you have, where it comes from, what you do with it, and where it goes. For AI agencies, data mapping must cover both traditional data flows and AI-specific data flows.
Traditional Data Mapping
Document every category of personal information your agency collects, receives, or processes. For each category, record the sources, the business purpose, the retention period, and the third parties with whom it is shared.
AI-Specific Data Mapping
In addition to traditional data mapping, AI agencies must map:
- Training data. What personal information is used to train models? Where does it come from? Is it covered by service provider agreements?
- Feature engineering data. What personal information is used to create features for model input? Are any features derived from sensitive personal information?
- Model inputs. What personal information flows into models during inference? Is it processed and discarded or retained?
- Model outputs. What inferences are generated? Which qualify as personal information? Where are they stored and shared?
- Evaluation data. What personal information is used to evaluate model performance? How is it managed after evaluation?
- Third-party AI tools. What personal information flows to third-party AI services (cloud ML platforms, API-based models, data enrichment services)? What are the contractual protections?
Data Inventory Template
For each AI project, maintain a data inventory that includes:
- Data category and specific data elements
- Source of data (client, public, third-party, generated)
- Business purpose for processing
- Legal basis (service provider agreement, consent, other)
- Whether data includes sensitive personal information
- Retention period and deletion procedures
- Third parties who receive the data
- Technical and organizational safeguards
- Whether data is used for model training, inference, or both
Implementing CCPA Compliance in Your AI Workflow
During Project Scoping
Before starting any AI project, assess the CCPA implications. Ask your client:
- Will the system process personal information of California residents?
- What categories of personal information will be involved?
- Does the client's privacy policy disclose the intended use of the data?
- Has the client implemented consumer rights request mechanisms?
- Are there opt-out signals (Global Privacy Control) that need to be honored?
- Will sensitive personal information be involved?
Document the answers. If the project involves personal information from California residents, ensure your service provider agreement is in place and covers the specific use case.
During Data Collection and Preparation
- Verify that data collection is consistent with disclosed purposes
- Apply data minimization—collect only what is necessary for the specified purpose
- Identify and flag sensitive personal information for additional protections
- Check for opt-out signals and honor them before data enters your pipeline
- De-identify or aggregate data where possible to reduce regulatory exposure
- Document data provenance for every dataset
During Model Development
- Limit training data to what is covered by your service provider agreement
- Do not combine personal information from different clients without explicit authorization
- Implement access controls that restrict who can access personal information during development
- Log all access to personal information during the development process
- Test models for disparate impact that could result in discriminatory outputs
During Deployment and Operations
- Implement data retention limits consistent with the specified business purpose
- Build deletion capabilities that can respond to consumer rights requests
- Monitor for data drift that could introduce new categories of personal information
- Ensure model outputs that constitute personal information (inferences) are handled in compliance with CCPA
- Maintain audit trails that document your compliance throughout the system lifecycle
Handling Consumer Rights Requests
When a client receives a consumer rights request that affects your AI systems, you must be prepared to respond. Build these capabilities before you need them.
Right to Know Requests
You must be able to identify all personal information you hold about a specific consumer across all systems, including training data, feature data, and model outputs. Build a consumer data lookup capability that can search across your data stores and return a complete picture.
Deletion Requests
You must be able to delete a specific consumer's personal information from your systems. This includes source data, processed data, and derived data. For AI models, determine whether deletion of source data is sufficient or whether model retraining is required. Document your approach and the reasoning behind it.
Correction Requests
You must be able to correct inaccurate personal information, including inferences generated by your models. If a consumer disputes an inference, you need a process for investigating and correcting it.
Response Timelines
Businesses must respond to consumer requests within 45 days, with a possible 45-day extension. As a service provider, you must assist your client in meeting these timelines. Build internal SLAs that give you enough time to fulfill your part while leaving your client time to compile and deliver the response.
Automated Decision-Making Under CCPA
The CPRA amendments added specific provisions for automated decision-making. Consumers have the right to opt out of automated decision-making technology, and the California Privacy Protection Agency is developing regulations that will further define the obligations around automated decision-making.
For AI agencies, this means:
- Disclose when your AI systems make automated decisions that produce legal or similarly significant effects
- Provide access to information about the logic involved in the decision-making process
- Build the ability for human review of automated decisions
- Enable consumers to opt out of automated decision-making where required
- Document your automated decision-making processes and their potential impacts
Enforcement and Penalties
The California Attorney General can bring civil actions for CCPA violations, with penalties of up to 2,500 dollars per unintentional violation and 7,500 dollars per intentional violation. The California Privacy Protection Agency (CPPA) has administrative enforcement authority. Consumers have a private right of action for data breaches involving certain categories of personal information, with statutory damages of 100 to 750 dollars per consumer per incident.
For AI agencies, the risk is amplified by scale. A model trained on data from 100,000 consumers that violates CCPA could theoretically result in penalties of 250 million dollars for unintentional violations or 750 million dollars for intentional violations. While enforcement at this scale is unlikely for most agencies, the per-consumer penalty structure creates significant exposure.
Building a CCPA Compliance Program
Governance Structure
Appoint a privacy lead who is responsible for CCPA compliance. For agencies under 25 people, this can be a senior leader with privacy as part of their portfolio. For larger agencies, consider a dedicated privacy professional or fractional data protection officer.
Policies and Procedures
Develop written policies covering data collection and use, consumer rights request handling, data retention and deletion, incident response for data breaches, vendor and sub-processor management, and employee training and awareness.
Training
Train all employees who handle personal information on CCPA requirements, your policies and procedures, and how to identify and escalate privacy issues. Conduct refresher training annually and whenever significant regulatory changes occur.
Monitoring and Auditing
Conduct regular audits of your data handling practices. Review service provider agreements annually. Monitor regulatory developments including new CPPA regulations. Update your compliance program based on audit findings and regulatory changes.
Your Next Step
This week: Create a data inventory for your three largest active projects. For each project, document the categories of personal information you process, the sources, the purposes, and the third parties involved. Identify whether any projects involve sensitive personal information.
This month: Review all client service provider agreements for CCPA compliance. Ensure each agreement includes the required CCPA provisions. Where gaps exist, work with your clients to update the agreements. Implement a consumer rights request response capability that can identify and extract a specific consumer's data across your systems.
This quarter: Build CCPA compliance into your standard project delivery workflow. Create templates and checklists that ensure every new project addresses CCPA requirements from day one. Conduct a comprehensive privacy audit of all active projects and remediate any issues identified. Train all team members on CCPA requirements and your compliance procedures.