An edtech company hired an AI agency in Seattle to build a personalized learning assistant for elementary school students in 2025. The AI tutor adapted to each student's learning pace, tracked their progress, and generated customized exercises. Parents loved it. Teachers praised it. Then the FTC came calling. The investigation revealed that the AI system was collecting detailed behavioral data from children under 13—interaction patterns, time-on-task metrics, error patterns, and free-text responses—without obtaining verifiable parental consent. The AI system was also using this data to improve its models, effectively using children's educational interactions as training data. The FTC settlement required the edtech company to pay $3.5 million in penalties, delete all data collected from children, and destroy any models trained on that data. The AI agency was not named in the enforcement action, but the edtech company sued the agency for breach of contract, alleging that the agency failed to advise on COPPA compliance during development.
If your agency builds AI systems that interact with children or process children's data, you are operating in one of the most enforcement-active areas of privacy law. The FTC has made children's privacy a top enforcement priority, and COPPA violations carry severe financial penalties plus the operational devastation of mandatory data and model deletion. This post covers what you need to know to build AI for children-facing applications without creating a compliance disaster.
COPPA Fundamentals
What COPPA Covers
The Children's Online Privacy Protection Act applies to operators of commercial websites and online services (including AI-powered applications) that are directed at children under 13, or that have actual knowledge that they are collecting personal information from children under 13.
"Directed at children" is determined by factors including:
- The subject matter of the site or service
- Visual content and design (cartoons, child-oriented characters)
- Language and vocabulary level
- The presence of child-oriented features
- Whether advertising targets children
- Whether the intended audience includes children
"Actual knowledge" means the operator knows that a specific user is under 13. This can come from age gates, user profiles, or contextual information.
What COPPA Requires
Verifiable parental consent: Before collecting personal information from a child under 13, you must obtain verifiable consent from the child's parent or guardian. The consent must be informed—the parent must understand what data is collected, how it is used, and who it is shared with.
Privacy notice: Provide a clear, comprehensive privacy notice describing your data collection and use practices.
Data minimization: Collect only the personal information reasonably necessary for the activity. Do not condition participation on providing more information than necessary.
Data security: Maintain reasonable procedures to protect the confidentiality, security, and integrity of children's personal information.
Data retention limits: Retain children's personal information only as long as reasonably necessary, then delete it securely.
Parental access and deletion rights: Parents have the right to review their child's personal information, request deletion, and refuse further collection.
What Counts as Personal Information Under COPPA
COPPA's definition of personal information is broad and covers data that AI systems commonly collect:
- Name, address, email, phone number: Obviously covered
- Screen name or username: Covered if it functions as an identifier
- Geolocation data: Covered
- Photos, videos, and audio recordings: Covered, including audio recordings of children speaking to AI assistants
- Persistent identifiers: Cookies, device IDs, IP addresses when used to recognize a user over time
- Information combined with any of the above: If you combine behavioral data with a persistent identifier, the behavioral data becomes personal information
AI-specific considerations: When a child types or speaks to an AI system, the text of their input is likely personal information under COPPA if it can be associated with a persistent identifier. This means chatbot conversations, voice assistant interactions, and free-text responses in educational AI tools are all subject to COPPA requirements.
AI-Specific COPPA Challenges
Conversational AI with Children
AI chatbots and voice assistants designed for children create particular COPPA challenges.
Input collection: Every message a child sends to an AI chatbot is potentially personal information. Children may share their name, location, school, family details, health information, or emotional state in conversation with an AI assistant.
Context retention: AI systems that maintain conversation context across sessions are storing children's personal information. Multi-turn conversations that build on previous interactions create growing data stores of children's communications.
Model training: If AI conversation data from children is used to improve models (through fine-tuning, RLHF, or other training methods), that constitutes a use of children's personal information that must be disclosed and consented to.
Output content: AI systems can generate content that is inappropriate for children—violent, sexual, or otherwise harmful content. While this is not strictly a COPPA issue, it is a child safety concern that regulators and parents will hold you accountable for.
Personalization and Profiling
AI systems that personalize experiences for children—adaptive learning platforms, recommendation engines, personalized content feeds—are building profiles of children.
The COPPA problem: Building a profile of a child's preferences, behaviors, and characteristics using persistent identifiers requires verifiable parental consent. If your AI system adapts to a child's learning style, tracks their progress, or personalizes content, you are almost certainly profiling under COPPA.
The data minimization problem: COPPA requires collecting only data necessary for the activity. If a child is using an educational AI tool to learn math, collecting data about their reading habits, social interactions, or emotional state may exceed what is necessary for the activity.
Behavioral Analytics
Many AI systems collect detailed behavioral data—click patterns, time on task, error rates, navigation paths, engagement metrics. When collected from children and associated with persistent identifiers, this behavioral data is personal information under COPPA.
Common failures:
- Analytics SDKs and tracking tools running on children's applications without COPPA-compliant configuration
- Behavioral data being sent to third-party analytics services without parental consent
- Detailed interaction logs being retained indefinitely when they should be deleted after a limited period
Voice and Image Data
AI systems that process children's voices (voice assistants, speech recognition, pronunciation assessment) or images (facial recognition, emotion detection, video tutoring) collect particularly sensitive data.
Voice recordings of children are personal information under COPPA. Storing, processing, or transmitting these recordings requires parental consent and appropriate safeguards.
Facial and image data of children is similarly covered. AI systems that use cameras to detect attention, emotion, or engagement are collecting children's biometric data—among the most sensitive categories of personal information.
The FTC's Enforcement Posture
The FTC has been aggressive in COPPA enforcement, particularly around AI and edtech applications.
Key enforcement trends:
- Mandatory deletion of models trained on children's data: The FTC has required companies to not only delete illegally collected children's data but also destroy any algorithms or models trained on that data. This "algorithmic disgorgement" remedy means that COPPA violations can destroy your AI investment, not just your data.
- Expanding the definition of directed at children: The FTC has taken a broad view of what constitutes a service "directed at children," including services that are not exclusively for children but that have significant child audiences.
- Third-party accountability: While COPPA primarily applies to operators (your clients), the FTC has signaled that third parties that facilitate COPPA violations may face scrutiny.
- Increased penalties: COPPA penalties have increased significantly, with recent settlements in the millions of dollars.
Governance Framework for Children's AI
Pre-Development Assessment
Before building any AI system that might involve children, conduct a thorough assessment.
Audience determination: Will the AI system be directed at children under 13? Could children under 13 use it even if it is not specifically directed at them? If the answer to either question is yes, COPPA applies.
Data mapping: Identify every type of data the AI system will collect, process, or store. For each data type, determine whether it constitutes personal information under COPPA.
Necessity assessment: For each data type, determine whether it is reasonably necessary for the activity. Document your justification for collecting each type of data.
Consent mechanism design: Design the verifiable parental consent mechanism. The FTC accepts several methods, including signed consent forms, credit card transactions, video conferencing, and knowledge-based authentication. Choose a method appropriate to your context.
Development Standards
Privacy by design: Build COPPA compliance into the architecture, not as an afterthought.
- Implement age gates or age verification mechanisms
- Build consent management into the user flow
- Design data collection to minimize what is gathered
- Implement data retention limits in the system design
- Build parental access and deletion capabilities from the start
Content safety: Implement safeguards to prevent AI systems from generating inappropriate content for children.
- Content filtering for AI outputs
- Topic restrictions appropriate for the child's age
- Monitoring and logging for review
- Escalation procedures for concerning interactions
Data isolation: Children's data should be isolated from adult data in your systems.
- Separate storage for children's personal information
- Separate access controls with stricter permissions
- Separate retention policies with shorter timeframes
- Clear labeling of children's data throughout the pipeline
Consent Management
Obtaining consent: Before collecting any personal information from a child, obtain verifiable parental consent.
- Provide a clear notice to the parent explaining what data is collected, why, and how it will be used
- Use an FTC-approved consent verification method
- Record the consent and maintain records of when and how consent was obtained
- Allow parents to consent to some uses but not others (for example, consent to data collection for the service but not for marketing)
Managing consent: After obtaining consent, maintain ongoing consent management.
- Allow parents to review their child's data at any time
- Allow parents to revoke consent at any time
- When consent is revoked, stop collecting data and delete existing data
- Notify parents of any material changes to data practices and obtain renewed consent
Ongoing Monitoring
Data collection monitoring: Regularly audit what data your AI system is collecting from children. Compare actual collection against your documented data map. Identify and address any gaps.
Retention monitoring: Verify that data retention limits are being enforced. Children's data should not persist beyond the documented retention period.
Third-party monitoring: If your AI system shares children's data with third parties (analytics services, cloud providers, model training platforms), verify that those third parties are handling the data in compliance with COPPA and your privacy notice.
Incident monitoring: Monitor for security incidents, data breaches, and inappropriate content generation. Have incident response procedures specific to children's data.
Contractual and Business Considerations
Client Contracts
When building children's AI for clients, your contracts should address:
- COPPA compliance responsibility allocation: Define who is responsible for which compliance activities. The operator (your client) bears primary COPPA responsibility, but your agency's role in system design and data handling creates shared obligations.
- Data handling requirements: Specify how children's data is handled, stored, and deleted throughout the engagement and after it ends.
- Consent mechanism requirements: Define the consent mechanisms and verify that they meet FTC standards before deployment.
- Audit rights: Allow for compliance audits of the AI system.
- Indemnification: Address liability for COPPA violations. Both parties should understand their exposure.
- Model training restrictions: Explicitly address whether children's data can be used for model training and under what conditions.
Insurance Considerations
Standard errors and omissions insurance may not cover COPPA violations or regulatory penalties. Review your insurance coverage with your broker and consider:
- Cyber liability coverage that includes regulatory penalties
- Specific coverage for data privacy violations
- Coverage for algorithmic disgorgement costs (the cost of rebuilding models if a regulator requires model destruction)
Pricing
COPPA-compliant AI development is significantly more expensive than non-regulated AI development. Price your engagements to cover:
- Privacy impact assessment and data mapping
- Consent mechanism development and implementation
- Content safety systems
- Parental access and deletion capabilities
- Ongoing monitoring and compliance management
- Legal review of privacy notices and consent mechanisms
- Separate data infrastructure for children's data
Beyond COPPA: Other Children's Privacy Requirements
State laws: Several states have enacted children's privacy laws that go beyond COPPA. California's AADC (Age Appropriate Design Code) imposes design requirements on services likely to be accessed by children. Other states have similar laws. Identify the states where your application will be available and comply with all applicable laws.
International laws: The UK's Age Appropriate Design Code, the EU's GDPR (with specific provisions for children's data), and other international laws may apply if your application is accessible outside the US.
Industry standards: The edtech industry has developed standards and certifications for children's privacy (such as Student Data Privacy Consortium pledges). Your clients may require compliance with these standards.
Platform requirements: App stores (Apple, Google) have specific requirements for children's applications that go beyond legal requirements. Familiarize yourself with these requirements and build compliance into your development process.
Your Next Step
If your agency builds or plans to build AI for children or educational applications, start with a COPPA compliance assessment. Review your current and planned projects to determine which ones involve children's data. For each project, map the data flows, identify COPPA-covered personal information, and assess whether your consent mechanisms, data handling practices, and retention policies meet COPPA requirements.
Then establish your children's AI governance framework: development standards, consent management procedures, monitoring protocols, and incident response plans. The cost of building this framework upfront is a fraction of the cost of an FTC enforcement action. And the agency that can demonstrate COPPA compliance maturity wins the trust of educational institutions, children's media companies, and family-focused brands that represent a significant and growing market for AI services.