Building an AI Incident Response Plan: When Your Model Goes Wrong

At 3:47 AM on a Tuesday, a pricing algorithm built by an AI agency for a regional airline went haywire. A data pipeline issue fed the model stale inventory data, and the model responded by dropping prices on 40 routes to $14 per ticket. By 6 AM, over 12,000 tickets had been sold at those prices. The airline's revenue management team discovered the problem when they arrived at work at 7:30 AM. They called the agency in a panic. The agency's first response was to ask "Who handles this?" Nobody knew. There was no incident response plan. The engineer who built the model was on vacation. The project manager had moved to a different team. Nobody could find the monitoring dashboard credentials. It took four hours just to identify the root cause and another three to deploy a fix. By then, the airline had absorbed $2.1 million in lost revenue. The relationship with the client never recovered.

Every AI system will eventually fail. Models drift. Data pipelines break. Edge cases emerge. Adversaries probe. The question is not whether your AI systems will have incidents — it's whether you'll be prepared when they do. An AI incident response plan is the structured process your agency follows when an AI system behaves in ways that cause harm, create risk, or deviate significantly from expected behavior. Without one, incidents become crises. With one, they become manageable events that you resolve quickly and learn from.

Why AI Incidents Are Different from Traditional Software Incidents

Traditional incident response plans are designed for server outages, security breaches, and software bugs. AI incidents share some characteristics with these traditional incidents but have unique properties that require specialized response procedures.

AI failures are often silent. A crashed server announces itself immediately. An AI model that starts producing biased results may do so gradually and invisibly. By the time someone notices, the damage has accumulated over days, weeks, or months.

AI failures can cause novel harms. Traditional software failures typically cause downtime or data loss. AI failures can cause discrimination, financial harm, physical safety risks, and reputational damage — harms that are harder to quantify and remediate.

Root cause analysis is harder. When a traditional software system fails, you can often trace the failure to a specific code change, configuration error, or infrastructure problem. AI failures may stem from subtle data distribution changes, unforeseen feature interactions, or emergent model behaviors that don't have a single identifiable cause.

Rollback is more complex. Rolling back a traditional software deployment restores the previous version's behavior exactly. Rolling back an AI model may not be straightforward if the previous version was also affected by the underlying issue (such as data drift) or if the rollback itself creates inconsistencies in downstream systems.

Remediation may require retraining. Fixing a traditional software bug means changing code. Fixing an AI issue may require retraining the model with corrected data, adjusted features, or modified constraints — a process that takes hours or days rather than minutes.

Impact assessment requires statistical analysis. Determining who was affected by a traditional software outage is usually straightforward — everyone who used the system during the outage. Determining who was affected by an AI bias issue requires statistical analysis across populations, time periods, and decision types.

Types of AI Incidents

Your incident response plan should cover the full range of AI incidents your systems might experience.

Performance Degradation Incidents

The model's accuracy, precision, recall, or other performance metrics have declined below acceptable levels.

• Causes: Data drift, concept drift, upstream data quality changes, infrastructure issues affecting inference
• Detection: Performance monitoring alerts, user complaints about quality, business metric anomalies
• Impact: Incorrect decisions, financial losses, reduced user trust
• Urgency: Depends on the severity of degradation and the stakes of the decisions affected

Fairness and Bias Incidents

The model is producing discriminatory outcomes for specific groups.

• Causes: Training data bias amplification, feature drift that affects groups differently, interaction between model updates and population changes
• Detection: Fairness monitoring alerts, complaints from affected individuals, external audits, media attention
• Impact: Discrimination against protected groups, regulatory violations, reputational damage, legal liability
• Urgency: High — fairness incidents often escalate quickly once detected, and regulatory timelines may apply

Safety Incidents

The model has caused or could cause physical, financial, or psychological harm.

• Causes: Edge cases not covered in training, adversarial manipulation, system integration failures, model behavior outside its operational domain
• Detection: Safety monitoring, user reports, downstream system alerts, physical world consequences
• Impact: Physical harm, financial loss, psychological distress, legal liability
• Urgency: Critical — safety incidents require immediate response to prevent further harm

Security Incidents

The AI system has been compromised, or its security has been breached.

• Causes: Adversarial attacks, prompt injection, model extraction, unauthorized access, data exfiltration
• Detection: Security monitoring, anomaly detection, access logs, third-party reports
• Impact: Data breach, model compromise, unauthorized access to systems, loss of intellectual property
• Urgency: Critical — security incidents require immediate containment and may trigger regulatory notification obligations

Data Incidents

Problems with the data feeding the AI system or the data it produces.

• Causes: Data pipeline failures, data quality issues, data poisoning, data sovereignty violations, unauthorized data access
• Detection: Data quality monitoring, pipeline alerts, audit findings
• Impact: Model performance issues, privacy violations, compliance failures
• Urgency: Depends on the type and severity — data sovereignty violations and privacy breaches are urgent

Reputational Incidents

The AI system has generated public attention that threatens the client's or agency's reputation.

• Causes: Any of the above incident types becoming public, media coverage, social media virality, advocacy group attention
• Detection: Media monitoring, social media monitoring, client notification
• Impact: Brand damage, loss of customer trust, regulatory scrutiny, business disruption
• Urgency: High — reputational incidents require fast, coordinated response to control the narrative

Building the Incident Response Plan

Component 1: Incident Classification

Define a classification scheme that helps your team assess the severity and urgency of each incident.

Severity levels:

• Critical — The incident is causing active harm to individuals, creating immediate legal or regulatory exposure, or affecting safety. Requires immediate response, executive notification, and potentially taking the system offline.
• High — The incident is causing significant negative impact but is not immediately dangerous. Requires same-day response and senior leadership notification.
• Medium — The incident is causing measurable but limited negative impact. Requires response within 48 hours and project lead notification.
• Low — The incident represents a deviation from expected behavior that has not yet caused measurable harm. Requires response within one week and team notification.

Classification criteria:

• Is anyone being harmed right now?
• Is the harm reversible?
• How many people are affected?
• Does the incident create regulatory notification obligations?
• Is the incident publicly visible?
• Can the system be rolled back without creating additional issues?

Component 2: Response Team and Roles

Define who is involved in incident response and what each person does.

Incident Commander. A senior team member who coordinates the overall response. They make decisions about severity classification, escalation, and communication. They don't fix the technical problem — they manage the process.

Technical Lead. The team member responsible for diagnosing the root cause and implementing the fix. This should be someone with deep knowledge of the specific AI system involved.

Communications Lead. The person responsible for coordinating communication with the client, affected individuals, regulators, and (if applicable) the media. They ensure messaging is accurate, timely, and consistent.

Data Lead. The person responsible for analyzing the incident's impact on data — including affected populations, the scope of harm, and evidence preservation.

Legal Advisor. The person who assesses legal implications, advises on regulatory notification obligations, and guides evidence preservation.

On-call rotation. Establish an on-call rotation so there's always someone available to respond to critical incidents outside business hours. The on-call person doesn't need to fix the problem — they need to assess severity and escalate appropriately.

Component 3: Response Procedures

Define step-by-step procedures for each severity level.

Immediate response (first hour for Critical/High):

1. Detect and confirm. Verify that the incident is real and assess initial severity.
2. Contain. If the system is causing active harm, take immediate containment action. This may mean taking the system offline, switching to a fallback system, routing decisions to human review, or blocking specific inputs. Err on the side of containment — a briefly offline system causes less damage than a system that's actively causing harm.
3. Notify. Alert the incident response team and client stakeholders appropriate to the severity level.
4. Preserve evidence. Before making changes, preserve logs, model outputs, data snapshots, and monitoring data. This evidence is essential for root cause analysis and may be legally required.
5. Activate the response team. Assemble the response team and assign roles.

Investigation (hours 1-24 for Critical/High):

1. Diagnose the root cause. Analyze the available evidence to determine what went wrong. For AI incidents, this often requires examining data distributions, model behavior over time, and upstream system changes.
2. Assess impact. Determine who was affected, how many people were affected, and the severity of the impact. For fairness incidents, this requires disaggregated analysis across relevant groups.
3. Identify remediation options. Develop options for fixing the immediate problem and preventing recurrence. For AI systems, remediation may include model rollback, retraining, threshold adjustment, feature modification, or data pipeline fixes.
4. Communicate with stakeholders. Provide updates to the client and other stakeholders at regular intervals. Share what is known, what is being done, and what the timeline looks like.

Remediation (days 1-7 for Critical/High):

1. Implement the fix. Apply the chosen remediation. Test the fix thoroughly before deploying it to production.
2. Verify the fix. Monitor the system after remediation to confirm that the issue is resolved and no new issues have been introduced.
3. Remediate harm. For individuals who were negatively affected by the incident, determine and implement appropriate remediation. This may include correcting decisions, providing compensation, or offering alternative processes.
4. Notify regulators. If the incident triggers regulatory notification obligations (data breach notification, adverse action correction, etc.), complete the required notifications within the applicable timeframes.

Post-incident (week 1-4 after resolution):

1. Conduct a post-incident review. Analyze the incident, response, and remediation to identify lessons learned. What caused the incident? Why wasn't it detected earlier? Was the response effective? What should change to prevent similar incidents?
2. Update the incident response plan. Incorporate lessons learned into the plan. Add new detection mechanisms, update response procedures, and refine classification criteria.
3. Implement preventive measures. Make systemic changes to prevent similar incidents. This may include additional monitoring, improved testing, architectural changes, or process improvements.
4. Close the incident. Document the final status, total impact, costs, and lessons learned. Update the incident register.

Component 4: Communication Plan

Define how and when you communicate about incidents.

Internal communication:

• Incident Commander provides regular updates to the response team (hourly for Critical, every 4 hours for High)
• Status page or shared document tracks the current state of the incident, actions taken, and next steps
• Leadership receives summary updates at defined intervals

Client communication:

• Initial notification within 1 hour for Critical incidents, 4 hours for High
• Regular updates at agreed intervals until resolution
• Final incident report within 2 weeks of resolution
• Template communications prepared in advance for common incident types

Regulatory communication:

• Assess notification obligations immediately upon incident classification
• Prepare regulatory notifications in consultation with legal counsel
• Submit notifications within required timeframes
• Document all regulatory communications

Public communication:

• Do not communicate publicly without coordination between the agency, the client, and legal counsel
• If the incident becomes public, prepare a statement that acknowledges the issue, describes the response, and outlines remediation
• Designate a single spokesperson for public communication

Component 5: Post-Incident Review Process

The post-incident review is where organizational learning happens. Do it well, and you prevent future incidents. Skip it, and you're destined to repeat the same failures.

Conduct the review within two weeks of incident resolution. Too soon, and emotions are still high. Too late, and memories have faded.

Include everyone involved in the response. Diverse perspectives reveal different aspects of what worked and what didn't.

Focus on systems, not individuals. The goal is to improve processes, tools, and procedures — not to assign blame. A blameless post-incident culture encourages honest analysis and genuine learning.

Produce a written report that includes:

• Incident timeline from detection through resolution
• Root cause analysis
• Impact assessment (who was affected, how, and for how long)
• Response assessment (what worked, what didn't, what was missing)
• Lessons learned
• Action items with owners and deadlines

Track action items to completion. Post-incident action items are easy to deprioritize when the urgency fades. Assign owners, set deadlines, and track completion.

Implementing the Plan in Your Agency

Make It Accessible

Your incident response plan is useless if nobody can find it when they need it. Store it in a location that's accessible to all team members, even outside business hours. Consider a dedicated page in your team wiki, a bookmark in your team communication tool, and a printed copy in the office.

Conduct Tabletop Exercises

Practice your incident response plan with simulated incidents. Tabletop exercises walk the team through a hypothetical scenario, testing whether everyone knows their role, whether the procedures work, and whether the communication channels function.

Run tabletop exercises quarterly. Vary the scenarios to cover different incident types: a performance degradation caught by monitoring, a bias complaint from a client, a security breach reported by a user, and a data pipeline failure at 2 AM.

Define Incident Response for Each Client

Your general incident response plan provides the framework. For each client, customize it with client-specific details:

• Client notification contacts and preferred communication channels
• Client-specific severity criteria (what constitutes a critical incident for this client?)
• System-specific rollback procedures
• Client-specific regulatory notification obligations
• Contractual requirements for incident response (response time SLAs, reporting requirements)

Maintain an Incident Register

Track all incidents across your portfolio in a central register. This provides:

• A record of all incidents for audit and compliance purposes
• Data for identifying patterns (are certain types of incidents recurring?)
• Evidence of your incident response capability for clients and regulators
• Input for continuous improvement of your processes

Incident Response as a Client Service

Incident response capability is a service you can offer to clients.

Managed incident response. For clients who don't have the technical capability to respond to AI incidents, offer a managed service where your agency provides first-line response, investigation, and remediation.

Incident response planning. Help clients build their own incident response plans, customized for their AI systems and organizational structure.

Incident response training. Train client teams to recognize, classify, and respond to AI incidents using their incident response plan.

Post-incident review facilitation. Facilitate post-incident reviews for clients, bringing an external perspective and structured methodology.

Your Next Steps

This week: Assess whether your agency has any documented incident response procedures for AI systems. If not, draft a basic plan using the framework in this guide.

This month: Conduct a tabletop exercise with your team using a realistic scenario. Identify gaps in your plan and your team's readiness.

This quarter: Implement a complete incident response plan, including on-call rotation, client-specific procedures, and an incident register. Conduct a second tabletop exercise to test the updated plan.

AI incidents are inevitable. The question is whether they're managed events that you resolve quickly and learn from, or unmanaged crises that damage your clients, your reputation, and your business. An incident response plan is the difference. Build one now, while things are calm, and you'll be ready when the next 3:47 AM call comes in.