A European insurance company deployed an AI claims processing system built by an agency in London. The system processed minor claims automatically, approving or denying them without human review. In early 2026, a policyholder whose claim was denied exercised their GDPR Article 22 right to challenge the automated decision and requested an explanation of why their claim was denied. The insurance company turned to the AI agency for help. The agency had built the system using a gradient-boosted tree ensemble with over 400 features. They could tell the policyholder that the model had scored their claim below the approval threshold. They could not tell the policyholder which specific factors most influenced the denial, why those factors led to denial, or what the policyholder could do differently. The insurance company faced a regulatory complaint, and the AI agency spent three months retrofitting explainability into a system that should have been designed with it from the start. The retrofit cost $95,000—more than the original build.
The right to explanation is no longer theoretical. It is embedded in laws across multiple jurisdictions, and enforcement is accelerating. If your agency builds AI systems that make decisions affecting individuals—credit decisions, insurance claims, hiring recommendations, content moderation, healthcare triage—those individuals may have a legal right to understand why the AI made the decision it did.
Building explainable AI is not just about slapping a SHAP plot on a dashboard. It requires designing systems from the ground up with explanation capabilities, understanding what different audiences need from explanations, and establishing governance around explanation quality and delivery.
The Legal Landscape for AI Explanations
GDPR Right to Explanation
GDPR Article 22 gives individuals the right not to be subject to decisions based solely on automated processing that produce legal effects or similarly significantly affect them. When automated decision-making is permitted (with consent, for contract performance, or by law), the data controller must provide "meaningful information about the logic involved."
What "meaningful information about the logic involved" means has been debated extensively since GDPR took effect. The current regulatory and judicial consensus is:
- The explanation must describe the factors that influenced the decision
- The explanation must be specific to the individual, not just a generic description of the model
- The explanation must be understandable to the individual (not just to data scientists)
- The explanation must be provided in a timely manner
EU AI Act Transparency Requirements
The EU AI Act imposes transparency obligations on high-risk AI systems that go beyond GDPR.
- Users must be informed that they are interacting with an AI system
- Deployers of high-risk AI must provide meaningful explanations of AI decisions
- Technical documentation must describe how the AI system works at a level that enables oversight
- Logs must be maintained that enable tracing of AI decisions after the fact
US Legal Developments
The US does not have a single federal right to explanation, but sector-specific requirements exist and state laws are expanding.
ECOA adverse action notices: Lenders must provide specific reasons for credit denials, including when AI makes the decision.
FCRA: The Fair Credit Reporting Act requires that consumers be informed of factors adversely affecting their credit scores.
State laws: Colorado's AI Act, Illinois' AI Video Interview Act, and other state laws include transparency or explanation requirements for AI decisions.
EEOC guidance: For AI in employment, the EEOC expects that employers can explain how AI tools make hiring recommendations.
Industry-Specific Requirements
Healthcare: Clinicians using AI for diagnosis or treatment recommendations need to understand the AI's reasoning to exercise clinical judgment.
Financial services: Regulators expect that financial institutions can explain AI-driven decisions to customers and to the regulators themselves.
Insurance: Claims decisions, underwriting decisions, and pricing decisions made or influenced by AI must be explainable to policyholders and regulators.
Types of Explanations
Different stakeholders need different types of explanations. A one-size-fits-all approach does not work.
Individual Explanations
These are explanations provided to the individual affected by the AI decision.
Requirements:
- Written in plain, non-technical language
- Specific to the individual's situation (not generic model descriptions)
- Identify the key factors that influenced the decision
- Ideally actionable—the individual should understand what they could change to receive a different outcome
- Timely—provided when the decision is communicated or shortly thereafter
Example for a credit denial: "Your application was denied primarily because: (1) Your debt-to-income ratio of 52% exceeds our threshold of 43%. (2) Your credit history length of 14 months is shorter than our minimum preferred history of 24 months. (3) You have two recent hard inquiries within the past 6 months."
Stakeholder Explanations
These are explanations provided to business stakeholders (your clients) who need to understand how the AI system works to make business decisions.
Requirements:
- Balance technical accuracy with accessibility
- Describe the model's general behavior and key decision drivers
- Include performance metrics and limitations
- Address fairness and bias considerations
- Support regulatory compliance discussions
Regulatory Explanations
These are explanations provided to regulators who are evaluating the AI system's compliance.
Requirements:
- Technically detailed and precise
- Document the model's methodology, data sources, and validation
- Include fairness testing results
- Describe the model's limitations and failure modes
- Provide evidence of ongoing monitoring and governance
Technical Explanations
These are explanations for data scientists and engineers who need to understand the system for maintenance, debugging, and improvement.
Requirements:
- Full technical detail
- Model architecture, training methodology, feature engineering
- Performance metrics across segments and edge cases
- Feature importance and interaction analysis
- Known failure modes and limitations
Explainability Techniques
Model-Agnostic Methods
These techniques work with any model regardless of architecture.
SHAP (SHapley Additive exPlanations): Assigns each feature a contribution value for a specific prediction. SHAP values show how much each feature pushed the prediction higher or lower relative to the baseline.
Strengths:
- Theoretically grounded (game theory)
- Provides both local (individual prediction) and global (model-wide) explanations
- Consistent and fair attribution of feature contributions
Limitations:
- Computationally expensive for large models
- Assumes feature independence (can be misleading when features are correlated)
- Requires careful interpretation—SHAP values show correlation, not causation
LIME (Local Interpretable Model-agnostic Explanations): Creates a simple, interpretable model (linear regression, decision tree) that approximates the complex model's behavior around a specific prediction.
Strengths:
- Intuitive explanations
- Works with any model
- Relatively fast
Limitations:
- Explanations are approximations that may not be faithful to the actual model behavior
- Sensitive to perturbation strategy
- Can produce inconsistent explanations for similar inputs
Counterfactual explanations: Answer the question "what would need to be different for the outcome to change?" For example: "If your debt-to-income ratio were below 43%, your application would have been approved."
Strengths:
- Highly actionable for individuals
- Intuitive and natural (people naturally think in counterfactuals)
- Do not require disclosing the model's internal workings
Limitations:
- Multiple valid counterfactuals may exist (which one to present?)
- May suggest changes that are impossible for the individual
- Generating counterfactuals can be computationally expensive
Inherently Interpretable Models
For applications where explainability is a hard requirement, consider using models that are inherently interpretable.
Linear and logistic regression: Feature weights directly show the direction and magnitude of each feature's influence. Easy to explain and audit.
Decision trees: Decision paths are visible and can be traced for individual predictions. The explanation is the sequence of conditions that led to the decision.
Rule-based systems: Explicit rules can be directly communicated as explanations.
Generalized additive models (GAMs): Show the individual contribution of each feature as a smooth function, combining interpretability with flexibility.
Trade-offs: Inherently interpretable models may have lower predictive performance than complex models. However, the performance gap is often smaller than expected, especially when features are well-engineered. For high-stakes decisions where explanation is legally required, the explainability benefit may outweigh the performance cost.
LLM-Specific Explanation Approaches
Large language models create unique explanation challenges because they are opaque and their decision processes are not well-understood.
Attention visualization: Shows which parts of the input the model attended to when generating output. Useful for understanding what the model focused on but does not explain why.
Chain-of-thought prompting: Asking the model to explain its reasoning before providing an answer. The explanation may not accurately reflect the model's actual computation process, but it can provide useful context.
Retrieval attribution: For RAG systems, identifying which retrieved documents contributed to the response. This is more tractable than explaining the model's reasoning and is often more useful to end users.
Confidence scoring: Providing confidence levels with LLM outputs helps users understand how certain the model is about its response.
Designing Explainable AI Systems
Architecture for Explainability
Build explanation capabilities into your system architecture from the start.
Explanation service: Create a dedicated service or module that generates explanations for model predictions. This service should:
- Accept a prediction and its inputs
- Generate explanations appropriate to the requested audience (individual, stakeholder, regulator)
- Cache explanations for retrieval
- Log explanation requests and delivery
Feature logging: Log the feature values used for each prediction. You cannot explain a prediction after the fact if you did not save the inputs.
Model versioning: Maintain model version information with each prediction. If someone requests an explanation months after a decision, you need to know which model version made the decision.
Explanation templates: Create templates for common explanation scenarios. Templates ensure consistency, compliance, and readability.
Explanation Quality
Not all explanations are equal. Evaluate your explanations against quality criteria.
Fidelity: Does the explanation accurately reflect the model's actual behavior? An explanation that sounds good but does not match what the model actually did is worse than no explanation.
Completeness: Does the explanation cover the most important factors? An explanation that highlights minor factors while omitting the primary driver is misleading.
Comprehensibility: Can the target audience understand the explanation? An explanation full of technical jargon is useless for a consumer.
Actionability: Does the explanation help the individual understand what they can do differently? Actionable explanations are more useful and more compliant than descriptive ones.
Consistency: Do similar cases receive similar explanations? Inconsistent explanations undermine trust and may indicate explanation quality problems.
Testing Explanations
Test your explanations with real users before deployment.
User comprehension testing: Present explanations to representative users and assess whether they understand the key points. If users consistently misunderstand the explanation, it needs to be revised.
Accuracy testing: Compare explanations against ground truth. For model-agnostic methods, verify that the explanation faithfully represents the model's behavior.
Legal review: Have legal counsel review explanation templates to ensure they meet regulatory requirements.
Adversarial testing: Test whether explanations could be used to game the system. If explanations reveal too much about the model's logic, adversaries could manipulate inputs to receive favorable decisions.
Governance Framework
Explanation Policy
Document your agency's approach to AI explanations.
- Which AI systems require explanation capabilities
- What types of explanations are provided to which audiences
- How explanation requests are handled and fulfilled
- How explanation quality is monitored and improved
- How explanations are documented and retained
Compliance Tracking
Track explanation-related compliance across your engagements.
- Which engagements have explanation requirements (legal, contractual, or best practice)
- What explanation capabilities are implemented for each engagement
- How many explanation requests have been received and fulfilled
- What the average response time for explanation requests is
- Whether any explanation requests have been denied and why
Continuous Improvement
Explanations should improve over time.
- Collect feedback from individuals who receive explanations
- Analyze patterns in explanation requests to identify common concerns
- Update explanation templates based on feedback and regulatory developments
- Retrain explanation models when the underlying AI model is updated
- Benchmark your explanations against industry best practices
Your Next Step
Audit your current AI deployments for explanation readiness. For each system, determine: Does this system make decisions that affect individuals? Is there a legal requirement for explanation? Can the system currently generate individual-specific explanations? If not, what would be needed to add explanation capability?
For systems that need explanation capability and do not have it, prioritize based on legal risk and client exposure. Start with the highest-risk system and implement explanation capabilities using the appropriate technique for that model architecture. Build the explanation architecture into your standard development process so that every new AI system ships with explanation capability from day one.
The agency that builds explainable AI is not just compliant—it builds AI that clients trust, regulators accept, and end users understand. That trust is the foundation of lasting client relationships and growing market share in an industry where transparency is increasingly the price of admission.