A Toronto AI agency built an automated claims processing system for an insurance company. The system processed auto insurance claims and recommended approval amounts. After 18 months of operation, the insurance company's internal audit team discovered that the system was systematically recommending lower approval amounts for claims from neighborhoods with high proportions of minority residents. The pattern had gone undetected because nobody was auditing the system's outputs for geographic disparities. The insurance regulator opened an investigation, and the company faced potential fair claims settlement violations. The agency was brought in to explain and remediate. The root cause was a feature that used average repair costs by zip code, which reflected a historical pattern where body shops in minority neighborhoods charged less, often because residents had less negotiating leverage. The AI system had learned and automated this inequality. An algorithmic audit conducted before deployment, or during the first year of operation, would have caught this pattern. The remediation cost $260,000 and the agency's reputation in the insurance vertical suffered lasting damage.
Algorithmic auditing is the systematic examination of AI systems to assess whether they operate as intended, comply with applicable requirements, and produce outcomes that are fair and appropriate. It is becoming mandatory in multiple jurisdictions and is increasingly expected by enterprise clients regardless of regulatory requirements. Your agency needs to understand what auditing standards exist, how to conduct audits that meet those standards, and how to build auditing into your delivery process.
The Algorithmic Auditing Landscape
The algorithmic auditing landscape is evolving rapidly. Multiple frameworks, standards, and regulatory requirements are converging toward a set of common expectations.
Regulatory Audit Requirements
EU AI Act. Requires conformity assessments for high-risk AI systems, including testing of accuracy, robustness, and cybersecurity. Providers must implement quality management systems and maintain technical documentation sufficient for auditing. Third-party auditing is required for certain high-risk categories.
NYC Local Law 144. Requires annual bias audits by independent auditors for automated employment decision tools. The audit must assess the tool's disparate impact on the basis of sex, race, and ethnicity.
Colorado AI Act. Requires deployers of high-risk AI systems to use reasonable care to protect consumers from known or reasonably foreseeable risks of algorithmic discrimination. Requires impact assessments that function as a form of self-audit.
Canada's Algorithmic Impact Assessment (AIA). Requires federal government institutions to conduct impact assessments for automated decision systems. The assessment determines the level of scrutiny and the safeguards required.
Singapore's Model AI Governance Framework. Provides guidance on algorithmic auditing including testing for fairness, transparency, and accountability. While voluntary, it is widely adopted in the Singapore market.
Industry Standards and Frameworks
NIST AI Risk Management Framework (AI RMF). Provides a comprehensive framework for managing AI risks, including assessment practices. The framework includes the Govern, Map, Measure, and Manage functions that align with auditing activities.
IEEE 7000 series. Includes standards for ethically aligned design and algorithmic bias considerations. IEEE 2089 addresses age-appropriate AI, and IEEE 7010 addresses well-being metrics.
ISO/IEC 42001. The international standard for AI management systems, which includes requirements for risk assessment, performance evaluation, and improvement that align with auditing practices.
OECD AI Principles. Established principles for trustworthy AI that many national regulations reference, including transparency, accountability, robustness, and fairness.
Emerging Audit Certification Programs
Several organizations are developing audit certification programs for AI systems.
- ForHumanity has developed certification schemes for EEOC compliance, EU AI Act compliance, and other regulatory frameworks
- Big Four accounting firms are developing AI audit practices
- Specialized AI audit firms are emerging with domain expertise in algorithmic assessment
Types of Algorithmic Audits
Different audit types serve different purposes. Your agency should understand when each type is appropriate.
Pre-Deployment Audit
Conducted before an AI system goes into production to assess whether it is fit for purpose.
Scope:
- System design review against requirements
- Training data quality and representativeness assessment
- Model performance validation across relevant populations
- Fairness and bias assessment
- Robustness and security assessment
- Documentation and governance review
When required:
- Before deploying any high-risk AI system
- When regulatory compliance must be demonstrated before operation
- When the client requires independent verification before go-live
Operational Audit
Conducted during the system's operation to assess ongoing performance and compliance.
Scope:
- Production performance monitoring review
- Drift detection and assessment
- Fairness metric tracking and trend analysis
- Incident review and remediation assessment
- Governance compliance verification
- Documentation currency review
When required:
- Periodically during operation, typically annually at minimum
- When triggered by performance anomalies or incidents
- When regulatory requirements specify ongoing assessment
Compliance Audit
Conducted to verify compliance with specific regulatory requirements.
Scope:
- Mapping system capabilities and practices to regulatory requirements
- Evidence gathering for compliance claims
- Gap identification and remediation planning
- Documentation preparation for regulatory submission
When required:
- When specific regulations apply such as EU AI Act conformity assessment or NYC Local Law 144
- When preparing for regulatory examination
- When client contracts require compliance certification
Impact Audit
Conducted to assess the real-world impact of the AI system on affected individuals and communities.
Scope:
- Outcome analysis across affected populations
- Disparate impact assessment
- Feedback from affected individuals and communities
- Cumulative impact assessment considering other systems
- Unintended consequence identification
When required:
- When the system affects decisions about individuals
- When community impact is a concern
- When cumulative impact of multiple AI systems needs assessment
The Algorithmic Audit Process
Phase 1: Audit Planning
Scope definition. Define what will be audited, what standards will be applied, and what constitutes a finding.
- Identify the AI system or systems to be audited
- Define the audit type: pre-deployment, operational, compliance, or impact
- Identify the applicable standards, regulations, and frameworks
- Define the audit criteria, including the specific requirements the system will be assessed against
- Define the audit timeline and resource requirements
- Identify the audit team and their qualifications
Evidence requirements. Define what evidence the audit team will need.
- System documentation including design specifications, data documentation, and model documentation
- Training data samples or statistics
- Model artifacts or access to the model for testing
- Production data samples or statistics
- Monitoring and alerting configuration and historical data
- Incident records and post-mortem reports
- Governance documentation including policies, procedures, and meeting minutes
- Access to system operators and stakeholders for interviews
Audit independence. Establish the independence of the audit team.
- For internal audits, the audit team should not include anyone who developed or operates the system
- For external audits, the audit firm should not have business relationships that compromise objectivity
- Document any potential conflicts of interest and the measures taken to address them
- The audit team should have the authority to access all relevant information and personnel
Phase 2: Evidence Gathering
Documentation review. Examine all available documentation about the system.
- Review system design documentation against stated requirements
- Review data documentation including data sheets, data dictionaries, and data quality reports
- Review model documentation including model cards, validation reports, and performance metrics
- Review governance documentation including policies, procedures, and approval records
- Identify gaps in documentation that limit the audit's completeness
Technical assessment. Conduct technical testing of the system.
- Performance testing. Evaluate model accuracy, precision, recall, and other relevant metrics using appropriate test data
- Fairness testing. Evaluate the system's outputs for disparate impact across protected groups
- Robustness testing. Test the system's behavior under adversarial inputs, data drift, and edge cases
- Explainability assessment. Evaluate the quality and consistency of the system's explanations
- Security assessment. Test for vulnerabilities that could compromise system integrity or data protection
Process assessment. Evaluate the organizational processes around the system.
- Review the development process including requirements gathering, data selection, model development, and validation
- Review the deployment process including testing, approval, and rollout procedures
- Review the monitoring process including metrics tracked, alert thresholds, and response procedures
- Review the governance process including oversight mechanisms, review cadence, and accountability structures
Stakeholder interviews. Gather qualitative information from people involved with the system.
- Interview system developers about design decisions, trade-offs, and known limitations
- Interview system operators about operational challenges and incident experiences
- Interview business stakeholders about the system's impact on business processes and outcomes
- Where feasible and appropriate, gather feedback from individuals affected by the system
Phase 3: Analysis and Assessment
Findings identification. Analyze the evidence to identify audit findings.
- Compare observed practices and outcomes against audit criteria
- Categorize findings by severity: critical, major, minor, and observation
- Critical findings: The system causes or is likely to cause significant harm, violates a regulatory requirement, or has a fundamental design flaw
- Major findings: The system has a significant gap in governance, documentation, or performance that should be addressed before or shortly after the audit
- Minor findings: The system has a gap that should be addressed but does not present an immediate risk
- Observations: Opportunities for improvement that are not compliance or performance gaps
Root cause analysis. For each finding, identify the root cause.
- Is it a design issue that requires system modification?
- Is it a data issue that requires data remediation?
- Is it a process issue that requires procedural changes?
- Is it a governance issue that requires organizational changes?
- Is it a documentation issue that requires updated documentation?
Risk assessment. Assess the risk associated with each finding.
- What is the likelihood that the finding will result in harm?
- What is the potential severity of the harm?
- Who would be affected?
- What is the regulatory risk?
- What is the reputational risk?
Phase 4: Reporting
Audit report structure:
- Executive summary. One-page overview of the audit scope, approach, key findings, and overall assessment
- Audit scope and methodology. Detailed description of what was audited, how, and against what criteria
- Findings. Detailed description of each finding including evidence, severity, root cause, and risk assessment
- Recommendations. Specific remediation recommendations for each finding, prioritized by risk
- Management response. The audited organization's response to each finding, including planned remediation actions and timelines
- Auditor qualifications. Documentation of the audit team's qualifications and independence
Report distribution and confidentiality:
- Define who receives the full report versus a summary
- Classify the report as confidential and implement appropriate handling controls
- For regulatory compliance audits, prepare a version suitable for regulatory submission
- For public disclosure requirements like NYC Local Law 144, prepare a public summary
Phase 5: Remediation and Follow-Up
Remediation tracking. Track remediation actions to completion.
- Assign each remediation action to a responsible owner with a deadline
- Track remediation progress in a centralized system
- Verify that remediation actions address the root cause, not just the symptom
- Close findings only after verification that the remediation is effective
Follow-up audit. Conduct a follow-up audit to verify remediation.
- Schedule follow-up based on finding severity: critical findings within 30 days, major findings within 90 days
- Verify that remediation actions were implemented as planned
- Verify that the remediation effectively addressed the finding
- Document the follow-up results
Building Audit-Ready AI Systems
The most efficient approach to algorithmic auditing is building systems that are audit-ready from the start.
Documentation-first development. Create documentation as you develop, not after.
- Write model cards as part of the development process
- Document data decisions as they are made
- Record design decisions and trade-offs in decision logs
- Maintain version-controlled documentation alongside code
Automated audit evidence. Automate the collection of audit evidence.
- Implement comprehensive logging that captures the information auditors need
- Automate performance metric calculation and recording
- Automate fairness metric calculation and recording
- Maintain automated testing suites that can be re-run during audits
Governance by design. Build governance into the system architecture.
- Implement access controls that enforce governance policies
- Build monitoring and alerting into the system from the start
- Design data pipelines with audit trails built in
- Create approval workflows that generate audit evidence automatically
Audit simulation. Periodically simulate audits to test your readiness.
- Conduct internal mock audits using the same process an external auditor would follow
- Identify documentation gaps and evidence collection challenges before a real audit
- Train your team on how to respond to audit requests
- Use mock audit findings to improve your governance practices
Selecting and Managing External Auditors
When external auditing is required or preferred, select and manage auditors carefully.
Auditor selection criteria:
- Technical expertise in AI and machine learning
- Experience auditing AI systems in your industry or domain
- Understanding of applicable regulations
- Independence and absence of conflicts of interest
- Professional certifications or accreditations relevant to AI auditing
- References from similar engagements
Engagement management:
- Define the audit scope and criteria clearly in the engagement agreement
- Provide the auditor with timely access to documentation, systems, and personnel
- Designate an internal liaison to coordinate with the audit team
- Review and respond to draft findings promptly
- Implement remediation actions within agreed timelines
Your Next Step
Select one AI system your agency has delivered and conduct an internal pre-deployment or operational audit using the process above. Start with the documentation review: can you produce all the documentation an auditor would request? If you cannot, that is your first finding and your first remediation action.
Then conduct the technical assessment: test the system for performance, fairness, and robustness using the evidence gathering process described above. Document your findings in an audit report format. Even if the findings are all positive, the exercise of conducting the audit will reveal gaps in your audit readiness that you can address before a real audit occurs.
The agencies that build auditing capabilities now will be the ones that can deliver high-risk AI systems to regulated clients. The ones that wait until auditing is mandated will scramble to catch up while their competitors are already certified and winning contracts.