A financial services AI agency deployed a credit risk scoring model for a mid-tier bank. The model had been performing well for eight months. On a Tuesday morning, the model began assigning abnormally high risk scores to a specific demographic cluster of applicants. By the time the issue was detected—through a customer complaint escalated to the bank's compliance team on Thursday afternoon—the model had incorrectly denied 340 credit applications over 52 hours. The agency had no automated anomaly detection on model outputs. No one was monitoring fairness metrics in real time. The incident response process was ad hoc—the engineer who received the escalation did not know who to notify, what authority they had to take action, or what the communication protocol was. It took an additional 12 hours to diagnose the root cause (a data pipeline change that altered the encoding of a key variable) and another 6 hours to deploy a fix. The bank filed a formal complaint with its regulator, the agency refunded 90,000 dollars in fees, and the relationship never fully recovered.
AI incidents are different from traditional software incidents. They can be subtle—a model that slowly drifts rather than suddenly crashes. They can be discriminatory—affecting specific populations while working fine for others. They can be invisible to standard monitoring that tracks uptime and latency but not output quality and fairness. Your incident management playbook must account for these differences.
AI Incident Categories
Performance Degradation
The model's accuracy, precision, recall, or other performance metrics decline below acceptable thresholds. This can happen gradually (model drift) or suddenly (data pipeline failure, upstream system change). Performance degradation may not be visible to end users immediately but erodes the value and reliability of the system over time.
Bias and Fairness Incidents
The model produces systematically unfair outcomes for a specific group. This may emerge from training data bias that was not detected pre-deployment, from changes in the data distribution, or from interactions between the model and the real-world context that were not anticipated during development.
Safety Incidents
The model produces outputs that cause or could cause harm to individuals. In healthcare, this could mean incorrect clinical recommendations. In autonomous systems, it could mean unsafe control decisions. In financial services, it could mean inappropriate risk assessments that deny people access to essential services.
Privacy Incidents
The model exposes personal information through its outputs, leaks training data through adversarial queries, or processes data in ways that violate privacy regulations or agreements.
Security Incidents
The model or its infrastructure is compromised by adversarial actors. This includes model theft, data exfiltration, adversarial manipulation of inputs or outputs, and unauthorized access to model endpoints.
Availability Incidents
The model or its supporting infrastructure becomes unavailable. While similar to traditional IT availability incidents, AI availability incidents may have unique characteristics such as graceful degradation (the model still responds but with reduced capability) or partial failures (the model works for some input types but not others).
The Incident Management Framework
Severity Classification
Define severity levels that account for AI-specific characteristics:
Severity 1 (Critical). The AI system is causing active harm or has the potential to cause serious harm. Examples: a medical AI providing incorrect treatment recommendations, a safety system failing to detect hazards, a model systematically denying services based on protected characteristics.
Severity 2 (Major). The AI system is significantly degraded or producing materially incorrect outputs. Examples: model accuracy drops below the minimum acceptable threshold, a fairness metric exceeds the acceptable range, a data breach affects the AI system.
Severity 3 (Moderate). The AI system is experiencing issues that affect quality but do not pose immediate harm. Examples: gradual performance degradation detected by monitoring, minor bias detected in a specific edge case, intermittent errors in model outputs.
Severity 4 (Minor). The AI system has a known issue with limited impact. Examples: a cosmetic issue in model outputs, a non-critical monitoring alert, a documentation discrepancy.
Roles in Incident Management
Incident Commander. The person who leads the incident response. They coordinate the response team, make decisions about containment and communication, and ensure the incident is resolved. For Severity 1 and 2 incidents, this should be a senior engineering leader. For Severity 3 and 4, a project lead can serve as incident commander.
Technical Lead. The engineer who leads the technical investigation and remediation. They diagnose the root cause, develop the fix, and verify the resolution.
Client Liaison. The person who communicates with the affected client throughout the incident. This is typically the account manager or engagement lead. They provide updates, manage expectations, and coordinate on client-side actions.
Communications Lead. For high-severity incidents, a person responsible for managing broader communications, including internal updates, regulatory notifications, and public statements if needed.
Scribe. A person who documents the incident response in real time, including decisions made, actions taken, and timeline of events. This documentation is critical for post-incident review and regulatory compliance.
The Incident Response Process
Phase 1: Detection (Target: Minutes, Not Hours)
The speed of detection determines the scale of impact. Build multiple detection layers:
Automated monitoring. Implement monitoring that goes beyond traditional IT metrics:
- Model performance metrics (accuracy, precision, recall) compared to baselines
- Output distribution monitoring (detecting shifts in the distribution of model predictions)
- Fairness metrics across demographic groups
- Input data quality metrics (detecting anomalies in incoming data)
- Latency and throughput metrics
- Error rate monitoring
Alert configuration. Configure alerts with thresholds that balance sensitivity and noise. For high-risk systems, err on the side of more alerts. For lower-risk systems, tune thresholds to reduce false positives. Ensure alerts reach the right people through the right channels.
Stakeholder reporting. Create easy channels for end users, client teams, and internal team members to report concerns. A simple form or email alias that goes directly to the on-call engineer. Do not require stakeholders to diagnose the issue—just make it easy to report that something seems wrong.
Proactive checks. Conduct regular proactive checks on production systems—daily for high-risk systems, weekly for others. These checks should include model output sampling, fairness metric review, and data quality assessment.
Phase 2: Triage (Target: Within 30 Minutes)
When an incident is detected, triage it immediately:
Confirm the incident. Verify that the reported issue is real, not a false alarm. Check monitoring data, review recent changes, and attempt to reproduce the issue.
Classify the severity. Assign an initial severity level based on the impact assessment. The severity may be adjusted as more information becomes available.
Assign the incident commander. Based on the severity, assign the appropriate incident commander.
Notify stakeholders. Send initial notifications to the response team, the affected client, and leadership (for Severity 1 and 2). The initial notification should include what is known, what is being done, and when the next update will be provided.
Phase 3: Containment (Target: Within 1 Hour for Severity 1)
Contain the incident to prevent further harm before fully diagnosing the root cause.
Containment options for AI systems:
- Rollback. Revert to a previous known-good model version. This is the fastest containment option if you maintain versioned model deployments.
- Disable. Take the AI system offline. This is appropriate when the system is causing active harm and rollback is not available.
- Degrade gracefully. Switch to a simpler model, a rule-based fallback, or human review. This maintains service availability while removing the problematic model.
- Add guardrails. Implement additional output filters or validation checks that catch and block problematic outputs. This is appropriate when the issue is limited to specific output patterns.
- Reduce scope. Limit the AI system to the subset of inputs or users that are not affected by the issue. This maintains partial service while protecting affected populations.
Choose the containment option that most effectively stops the harm while minimizing disruption.
Phase 4: Investigation (Parallel With Containment)
Investigate the root cause of the incident. AI incidents often have non-obvious root causes that require systematic investigation.
Common root causes for AI incidents:
- Data pipeline changes that alter input features
- Data quality degradation in upstream sources
- Model drift due to changing data distributions
- Training data issues that were not detected pre-deployment
- Configuration errors in model serving infrastructure
- Third-party API changes that affect model inputs
- Edge cases not covered by testing
- Feedback loops that amplify model errors
- Adversarial manipulation of inputs
Investigation techniques:
- Compare recent model outputs to historical baselines
- Examine input data for anomalies or distribution shifts
- Review recent changes to the system, data pipeline, and infrastructure
- Test the model with known inputs to verify expected behavior
- Analyze error patterns to identify affected populations or input types
- Review logs for unusual activities or error messages
Phase 5: Resolution
Once the root cause is identified, implement a fix.
Fix development. Develop the fix in a controlled environment. For AI systems, the fix may involve retraining the model, adjusting the data pipeline, updating configurations, or implementing additional controls.
Fix testing. Test the fix thoroughly before deploying to production. Verify that it resolves the root cause, does not introduce new issues, and meets all acceptance criteria including fairness and performance requirements.
Fix deployment. Deploy the fix to production using your standard deployment process. Monitor closely after deployment to verify the fix is effective.
Impact remediation. Address the impact of the incident on affected parties. This may include reversing incorrect decisions, correcting erroneous outputs, or providing compensation.
Phase 6: Communication
Communicate throughout the incident response, not just at the end.
During the incident: Provide regular updates to the client, internal stakeholders, and leadership. Update frequency depends on severity—every 30 minutes for Severity 1, every 2 hours for Severity 2, daily for Severity 3.
After resolution: Provide a detailed incident report that covers what happened, why it happened, what was done, and what will be done to prevent recurrence.
Regulatory notifications: If the incident triggers regulatory notification requirements (such as HIPAA breach notification, GDPR data breach notification, or EU AI Act serious incident reporting), ensure notifications are made within the required timelines.
Phase 7: Post-Incident Review
Conduct a blameless post-incident review (postmortem) for all Severity 1 through 3 incidents.
Timeline reconstruction. Reconstruct a detailed timeline of the incident from detection through resolution.
Root cause analysis. Identify the root cause and contributing factors. Use techniques like the Five Whys or fishbone diagrams to get beyond surface-level causes.
Response evaluation. Evaluate the effectiveness of the incident response. Was detection fast enough? Was containment effective? Were communications timely and accurate?
Action items. Identify specific actions to prevent recurrence and improve the incident response process. Assign owners and deadlines.
Knowledge sharing. Share the post-incident review findings with the broader team. Anonymize client-specific details if needed, but share the lessons learned.
Building Your Incident Management Capabilities
On-Call Rotation
Implement an on-call rotation for your production AI systems. The on-call engineer should have the authority and capability to take containment actions without waiting for management approval. Define clear escalation procedures for situations that exceed the on-call engineer's authority.
Runbooks
Create runbooks for common incident types. A runbook is a step-by-step procedure for diagnosing and resolving a specific type of incident. Runbooks reduce response time and ensure consistent response regardless of which engineer is on call.
Incident Simulation
Conduct regular incident simulation exercises (game days) to test your incident response procedures, train your team, and identify gaps. Simulate realistic AI-specific scenarios—not just infrastructure failures, but bias incidents, model drift, and data quality issues.
Incident Tracking
Track all incidents in a centralized system. For each incident, record the detection time, triage time, containment time, resolution time, root cause, impact, and action items. Use this data to measure and improve your incident response performance over time.
Metrics for Incident Management
- Mean time to detect (MTTD). Average time from incident occurrence to detection. Target: under 30 minutes for Severity 1, under 4 hours for Severity 2.
- Mean time to contain (MTTC). Average time from detection to containment. Target: under 1 hour for Severity 1, under 4 hours for Severity 2.
- Mean time to resolve (MTTR). Average time from detection to full resolution. Track by severity level and trend over time.
- Incident rate. Number of incidents per system per quarter. Track trends to identify improvement or degradation.
- Post-incident review completion. Percentage of eligible incidents that receive a post-incident review. Target: 100 percent for Severity 1 through 3.
- Action item closure rate. Percentage of post-incident review action items closed within their deadlines.
Your Next Step
This week: Review your current incident detection capabilities for production AI systems. Identify the gaps—are you monitoring model performance? Output distributions? Fairness metrics? Identify the systems that would benefit most from enhanced monitoring.
This month: Implement monitoring and alerting for your highest-risk production AI systems. Define your severity classification system and incident response roles. Create an on-call rotation. Draft incident response procedures and communicate them to your team.
This quarter: Build out your full incident management program including runbooks for common incidents, incident simulation exercises, a centralized incident tracking system, and a post-incident review process. Train your team on incident response procedures. Establish incident metrics and begin tracking them.