A healthcare AI agency in Boston deployed a symptom assessment chatbot for a telemedicine platform in mid-2025. The chatbot had been tested against standard medical question-answer benchmarks and performed well. Two months after launch, a user described symptoms of a heart attack using non-standard language—chest tightness described as "feels like someone sitting on my chest" combined with jaw pain described as "my teeth hurt." The chatbot assessed the symptoms individually rather than recognizing the pattern, recommended dental care for the jaw pain, and suggested the user monitor the chest tightness. The user followed the chatbot's advice instead of calling emergency services. The user survived, but the incident triggered a malpractice investigation, a product liability claim against the telemedicine platform, and the immediate shutdown of the chatbot. The AI agency's testing had evaluated accuracy on standard medical queries but had not systematically evaluated safety—the system's behavior in scenarios where incorrect outputs could cause harm.
Safety evaluation is not the same as quality evaluation. Quality evaluation asks "does the AI produce good outputs?" Safety evaluation asks "can the AI produce outputs that cause harm?" These are fundamentally different questions that require different testing approaches, different metrics, and different decision frameworks.
This post provides a comprehensive safety evaluation framework for AI agencies—one that systematically identifies safety risks, tests for them, and provides clear criteria for deployment decisions.
The Difference Between Quality and Safety
Why Quality Testing Is Insufficient
Quality testing evaluates whether the AI system performs its intended function well. It measures accuracy, relevance, coherence, and user satisfaction on representative inputs. Quality testing answers the question: "How well does this system work in normal conditions?"
Safety testing evaluates whether the AI system can cause harm. It measures behavior on adversarial inputs, edge cases, failure modes, and high-stakes scenarios. Safety testing answers a different question: "How does this system behave in conditions where errors have consequences?"
The gap matters because:
- A system can be high-quality on average but dangerous in specific scenarios
- Quality metrics mask safety-critical failures by averaging them away
- Normal test distributions do not represent the long tail of inputs where safety failures occur
- Quality testing does not test for adversarial manipulation, which is how many real-world safety failures happen
Categories of AI Safety Risk
Physical safety: AI outputs that could lead to physical harm. Healthcare AI giving dangerous medical advice, autonomous systems making unsafe decisions, industrial AI providing incorrect operational parameters.
Psychological safety: AI outputs that could cause psychological harm. AI generating disturbing or traumatic content, AI systems that manipulate user emotions, chatbots that provide inappropriate responses to users in crisis.
Financial safety: AI outputs that cause financial harm through incorrect predictions, recommendations, or decisions. Trading algorithms, pricing models, fraud detection systems, credit scoring.
Information safety: AI outputs that spread misinformation, reveal private information, or provide information that could be used for harm. Hallucinated facts presented as truth, personal data leakage, instructions for dangerous activities.
Societal safety: AI systems that cause harm at a societal level through bias amplification, polarization, discrimination, or erosion of trust in institutions.
Security safety: AI systems that can be manipulated through prompt injection, adversarial inputs, or other attacks to produce unauthorized or harmful outputs.
The Safety Evaluation Framework
Phase 1: Risk Identification
Before testing, identify the safety risks specific to your AI system.
Stakeholder harm analysis: For each stakeholder who interacts with or is affected by the AI system, identify how the system could cause harm.
- Direct users: What happens if the AI gives them incorrect or harmful information?
- Subjects of AI decisions: What happens if the AI makes biased or unfair decisions about them?
- Third parties: Could the AI's outputs affect people who are not direct users?
- Society: Could widespread use of the AI cause societal harm?
Failure mode analysis: Identify the ways the AI system can fail and the consequences of each failure mode.
- What happens when the AI encounters inputs outside its training distribution?
- What happens when the AI encounters adversarial inputs?
- What happens when the AI's confidence is miscalibrated (it is confident about incorrect outputs)?
- What happens when the AI hallucinates?
- What happens when the AI is asked to do something outside its designed scope?
- What happens when upstream data sources fail or provide incorrect data?
Misuse analysis: Identify how the AI system could be intentionally misused.
- Can users manipulate the AI to produce harmful outputs through prompt injection?
- Can users use the AI to generate harmful content (misinformation, harassment, illegal content)?
- Can users exploit the AI to bypass safety controls in other systems?
- Can the AI be used for purposes it was not designed for that could cause harm?
Environmental analysis: Identify environmental factors that could create safety risks.
- Under what conditions might the AI's performance degrade?
- Are there seasonal, temporal, or situational factors that could trigger safety-relevant behavior changes?
- How does the AI behave under load or resource constraints?
Phase 2: Safety Test Design
For each identified risk, design tests that evaluate the AI's behavior.
Red team testing: Engage team members or external testers to actively try to make the AI produce harmful outputs. Red teamers should:
- Try to elicit harmful content through creative prompting
- Test edge cases and boundary conditions
- Attempt prompt injection and jailbreaking
- Test with inputs representing diverse populations and perspectives
- Try to exploit the system for unintended purposes
Adversarial testing: Systematically craft inputs designed to trigger safety failures.
- Inputs that are similar to safe inputs but include subtle modifications
- Inputs that exploit known weaknesses of the model architecture
- Inputs that test the boundaries of content policies
- Inputs in multiple languages and cultural contexts
Stress testing: Test the AI under conditions that may degrade safety.
- High load conditions
- Degraded input quality
- Partial system failures
- Unusual input distributions
- Extended interaction sequences
Scenario testing: Create realistic scenarios where safety failures would have consequences and evaluate the AI's behavior in those scenarios.
- Medical emergency scenarios for healthcare AI
- Financial distress scenarios for financial AI
- Crisis situations for customer service AI
- High-stakes decision scenarios for decision-support AI
Bias and fairness testing: Test for disparate performance across demographic groups that could constitute safety-relevant bias.
- Performance parity across racial and ethnic groups
- Performance parity across genders
- Performance parity across age groups
- Performance parity across ability levels
- Intersectional analysis
Phase 3: Safety Metrics
Define metrics that quantify safety performance.
Harm rate: The percentage of interactions where the AI produces outputs that could cause harm. This requires defining what constitutes "harm" for your specific system.
Safety refusal rate: The percentage of harmful requests that the AI correctly refuses. Higher is generally better, but excessive refusal (refusing safe requests) is also a problem.
False refusal rate: The percentage of safe requests incorrectly refused. Excessive false refusal degrades usability and can itself cause harm (imagine a medical AI refusing to discuss symptoms).
Adversarial robustness: The percentage of adversarial attacks that the AI correctly handles (refusing the request or producing a safe output despite the attack).
Confidence calibration: How well the AI's stated confidence matches its actual accuracy. Overconfident incorrect outputs are a significant safety concern.
Escalation rate: The percentage of interactions appropriately escalated to human review. A system that never escalates may be missing safety-critical situations.
Demographic parity: The variation in safety-relevant metrics across demographic groups. Large variations indicate safety-relevant bias.
Phase 4: Safety Decision Framework
Use test results to make deployment decisions.
Safety gates: Define pass/fail criteria for each safety metric. If the AI system fails any safety gate, it does not deploy until the failure is addressed.
Mandatory safety gates (examples):
- Harm rate below defined threshold (the threshold depends on the application domain)
- No systematic bias in safety performance across demographic groups
- Adversarial robustness above defined threshold
- All critical scenarios passed (100 percent safety on identified critical scenarios)
- Appropriate escalation behavior verified
Conditional safety gates: These gates may be passed with mitigation measures in place.
- Elevated harm rate in specific scenarios if those scenarios are addressed with human-in-the-loop review
- Moderate adversarial vulnerability if monitoring and rapid response are in place
- Some demographic performance variation if active monitoring and remediation are planned
Deployment decision matrix:
- All gates passed: Deploy with standard monitoring
- Some conditional gates with mitigations: Deploy with enhanced monitoring and defined mitigation measures
- Any mandatory gate failed: Do not deploy. Remediate and re-test.
- Multiple gates failed: Major redesign may be required. Reassess the approach.
Phase 5: Ongoing Safety Monitoring
Safety evaluation does not end at deployment. Ongoing monitoring catches safety issues that pre-deployment testing missed.
Production safety monitoring:
- Track harm rate in production using automated detection and human review samples
- Monitor for new adversarial attack patterns
- Track safety refusal rates and investigate changes
- Monitor user feedback for safety-relevant complaints
- Run periodic red team exercises on the production system
Safety incident response:
- Define what constitutes a safety incident
- Establish immediate response procedures (system suspension, mitigation, investigation)
- Track safety incidents and analyze them for patterns
- Feed safety incident findings back into the testing framework
Model update safety testing:
- When the underlying model is updated (by your agency or by a provider), re-run safety tests
- Compare safety metrics before and after the update
- Do not deploy updates that degrade safety performance without explicit review and approval
Safety Evaluation for Specific AI Types
Conversational AI (Chatbots, Virtual Assistants)
Specific safety concerns:
- Generating harmful advice (medical, legal, financial)
- Responding inappropriately to users in crisis (suicidal ideation, domestic violence)
- Producing discriminatory or offensive content
- Leaking personal information from training data or conversation context
- Being manipulated through prompt injection
Specific safety tests:
- Crisis scenario testing with standardized crisis prompts
- Content policy boundary testing
- Personal information probing
- Multi-turn conversation safety (does safety degrade over long conversations?)
- Language and dialect safety parity
Decision-Support AI (Recommendations, Scoring, Classification)
Specific safety concerns:
- Biased decisions affecting protected groups
- Incorrect high-confidence decisions in high-stakes contexts
- Failure to flag uncertain or edge cases for human review
- Gaming and manipulation by users who understand the scoring logic
Specific safety tests:
- Disparate impact testing across all protected groups
- High-stakes scenario accuracy testing
- Confidence calibration testing
- Adversarial input testing for decision manipulation
- Boundary condition testing (inputs near decision thresholds)
Generative AI (Content Generation, Code Generation)
Specific safety concerns:
- Generating misinformation or factually incorrect content
- Generating copyrighted or trademarked content
- Generating harmful instructions or content
- Generating content that impersonates real individuals
- Generating code with security vulnerabilities
Specific safety tests:
- Factual accuracy testing against verified knowledge
- Copyright and trademark detection testing
- Harmful content generation testing (CBRN, weapons, exploitation)
- Impersonation resistance testing
- Security vulnerability testing for generated code
Documentation and Governance
Safety Evaluation Documentation
Document your safety evaluation comprehensively.
Safety evaluation plan: Before testing, document your risk identification, test design, metrics, and decision criteria.
Safety evaluation report: After testing, document the results, the decision, and any mitigation measures.
Safety monitoring plan: Document your ongoing monitoring approach, metrics, thresholds, and response procedures.
Safety incident log: Maintain a log of all safety incidents, investigations, and corrective actions.
Safety Review Board
For agencies working on high-stakes AI, establish a safety review board that reviews safety evaluations before deployment decisions.
Board composition:
- Technical lead familiar with the AI system
- Governance or compliance representative
- Domain expert (healthcare professional, financial compliance expert, etc.)
- External advisor (for the highest-stakes systems)
Board authority:
- Can approve deployment, require additional testing, or block deployment
- Reviews safety evaluation reports and mitigation plans
- Reviews safety incident reports and corrective actions
- Sets and updates safety standards for the agency
Your Next Step
Select your highest-risk deployed AI system and conduct a safety evaluation using this framework. Start with risk identification—spend an hour brainstorming every way the system could cause harm. Then design tests for the top five identified risks and run them. Compare the results against your safety expectations.
You will likely find safety gaps you did not know existed. That is the point. Better to find them through systematic evaluation than through a production incident. Build safety evaluation into your development process for all new AI systems, and conduct periodic safety re-evaluations of deployed systems.
The agency that systematically evaluates AI safety deploys systems with confidence, responds to safety questions with evidence, and avoids the catastrophic incidents that destroy client relationships and agency reputations. Safety evaluation is not overhead—it is the foundation that everything else stands on.