New York City's Local Law 144 went into effect requiring employers using automated employment decision tools to conduct annual bias audits and publish the results. A staffing agency that had been using an AI-powered resume screening tool for two years scrambled to comply. They had never conducted a formal impact assessment. They had no documentation on how the tool made decisions, no demographic data to perform a bias audit, and no process for producing the required public summary. The compliance effort took four months and cost over $200,000 between legal fees, external auditor costs, and internal engineering time to extract the necessary data. When the audit was complete, it revealed statistically significant disparate impact against women in technical roles—a problem that had been baked into the system since deployment. The tool was shut down, the staffing agency faced reputational damage, and the AI vendor that built the tool lost multiple clients. All of this was preventable. An algorithmic impact assessment performed before deployment would have caught the bias issue, and an ongoing reporting process would have cost a fraction of the emergency compliance effort.
Algorithmic impact reports—also called algorithmic impact assessments (AIAs)—are structured evaluations of how an AI system affects the people and communities it touches. They are becoming a regulatory requirement in multiple jurisdictions, and even where they are not required, they are rapidly becoming an expectation from enterprise clients, investors, and the public.
What Algorithmic Impact Reports Cover
An algorithmic impact report evaluates an AI system across several dimensions of impact. It is broader than a technical model evaluation and broader than a legal compliance review. It asks: What are the real-world consequences of this system for the people it affects?
Fairness and bias impact. Does the system produce different outcomes for different demographic groups? Are these differences justified by legitimate factors, or do they reflect bias? This is the most legally scrutinized dimension and the one most frequently required by regulation.
Privacy impact. How does the system collect, use, and store personal data? Are data subjects aware of and consenting to this use? What privacy risks does the system create?
Transparency impact. Can affected individuals understand how the system makes decisions about them? Can they access meaningful explanations? Is the system's role in decision-making disclosed?
Accountability impact. Who is responsible when the system makes an error or causes harm? Are there appeal mechanisms? Is there human oversight?
Safety impact. Could the system cause physical, psychological, or financial harm? What are the failure modes and their consequences?
Social impact. Does the system affect access to opportunities, resources, or services? Does it concentrate or distribute power? Does it affect vulnerable populations disproportionately?
Economic impact. Does the system affect employment, wages, or economic opportunity? Does it create or destroy jobs? Does it affect market competition?
When Algorithmic Impact Reports Are Required
The regulatory landscape is evolving rapidly. Here are the key requirements as of 2026:
EU AI Act. High-risk AI systems must undergo a conformity assessment that includes evaluation of training data, system design, accuracy, robustness, and cybersecurity. Deployers of high-risk AI systems must conduct a fundamental rights impact assessment before putting the system into use.
New York City Local Law 144. Employers using automated employment decision tools must conduct annual independent bias audits and publish summary results.
Colorado AI Act. Deployers of high-risk AI systems must complete impact assessments evaluating the purpose, intended use, and potential discriminatory impact of the system.
Canada's AIDA (Artificial Intelligence and Data Act). Requires impact assessments for high-impact AI systems, including evaluation of potential biases and mitigation measures.
Illinois AI Video Interview Act. Companies using AI to analyze video interviews must notify candidates and obtain consent.
Various state and local proposals. Multiple US states and municipalities have proposed or are developing AI impact assessment requirements. The trend is clearly toward more requirements, not fewer.
Even without legal requirements, enterprise clients in financial services, healthcare, government, and other regulated sectors increasingly require algorithmic impact assessments as part of vendor selection and ongoing governance.
Building an Algorithmic Impact Report
Phase 1: System Description
Start with a clear, accessible description of the AI system. This section should be understandable by non-technical stakeholders.
System overview:
- What does the system do? Describe its function in plain language
- Who uses the system? Identify the deploying organization and the users
- Who is affected by the system? Identify the individuals and groups whose lives are impacted by the system's outputs
- What decisions does the system make or inform? Be specific about whether the system makes autonomous decisions or provides recommendations to human decision-makers
- What is the scale of deployment? How many people are affected and how frequently?
Technical summary:
- What type of AI/ML approach does the system use?
- What data does the system use to make decisions?
- How was the system trained and validated?
- What is the system's accuracy and error rate?
- What are the known limitations of the system?
Deployment context:
- In what organizational context is the system deployed?
- What process existed before the AI system was introduced?
- How does the AI system interact with human decision-makers?
- What safeguards and controls are in place?
Phase 2: Stakeholder Identification and Engagement
Identify everyone who is affected by the system and, where feasible, engage them in the assessment process.
Direct stakeholders are individuals the system makes decisions about or provides services to. For a hiring AI, this is job applicants. For a lending AI, this is loan applicants. For a healthcare AI, this is patients.
Indirect stakeholders are individuals affected by the system's broader impacts. For a hiring AI, this might include current employees whose roles are affected by automated screening, or communities that benefit or suffer from the hiring patterns the system produces.
Organizational stakeholders are the deployers, operators, and overseers of the system.
Engagement approaches:
- Surveys and interviews with affected individuals
- Focus groups with representative stakeholders
- Public comment periods for high-impact systems
- Advisory panels with community representatives
- Review of complaints and feedback from existing systems
Stakeholder engagement is not just a box to check. It surfaces impacts that technical analysis alone would miss. People affected by AI systems often have insights about real-world consequences that data scientists and engineers cannot anticipate.
Phase 3: Fairness and Bias Assessment
This is the most technically rigorous and legally consequential section of the report.
Define protected categories. Based on applicable laws and ethical principles, identify the demographic categories across which you will assess fairness. Common categories include race, ethnicity, gender, age, disability status, and socioeconomic status. The specific categories depend on the use case and jurisdiction.
Collect or estimate demographic data. To assess disparate impact, you need demographic data for the population affected by the system. This can come from:
- Self-reported demographic data (where available and appropriate)
- Proxy estimation using validated methods (Bayesian Improved Surname Geocoding, for example)
- Census or population-level demographic data for the relevant geography
- Third-party demographic data services
Conduct quantitative bias analysis:
- Selection rate analysis: Compare the rate at which the system produces favorable outcomes across demographic groups. The four-fifths rule (used in US employment law) says that if the selection rate for a protected group is less than 80% of the rate for the most favored group, there is evidence of adverse impact
- Statistical significance testing: Determine whether observed differences are statistically significant or could be explained by random variation
- Intersectional analysis: Assess bias at the intersection of multiple categories (e.g., Black women, elderly Asian men) because bias can be invisible at the aggregate level but present at intersections
- Error rate analysis: Compare error rates (false positives and false negatives) across demographic groups. Even if selection rates are equal, different error rates indicate different quality of service
Assess qualitative fairness dimensions:
- Procedural fairness: Is the process by which the system makes decisions fair and consistent?
- Distributive fairness: Are the system's outcomes distributed fairly across groups?
- Representational fairness: Are all groups adequately represented in the system's training data and evaluation?
- Recognition fairness: Does the system respect the dignity and identity of all individuals?
Document findings honestly. If the analysis reveals bias, document it clearly. Do not hide behind statistical hedging or minimize findings. The purpose of the assessment is to identify problems so they can be fixed.
Phase 4: Risk Assessment
Evaluate the risks the system creates across multiple dimensions.
Risk identification:
- What could go wrong? Brainstorm failure modes, misuse scenarios, and unintended consequences
- How likely is each risk? Assess probability based on available evidence
- How severe would each risk be? Assess impact on affected individuals and the organization
- Who bears the risk? Identify which stakeholders are most affected by each risk
Risk categories to evaluate:
- Discrimination risk: Could the system unlawfully discriminate?
- Privacy risk: Could the system expose personal information?
- Safety risk: Could the system cause physical or financial harm?
- Manipulation risk: Could the system be used to manipulate individuals?
- Surveillance risk: Could the system enable inappropriate surveillance?
- Autonomy risk: Could the system undermine individual autonomy or choice?
- Access risk: Could the system create barriers to essential services?
- Accuracy risk: Could the system make consequential errors?
Risk scoring:
For each identified risk, assign a score based on likelihood and severity. Use a standard risk matrix:
- Critical: High likelihood, high severity—immediate action required
- High: Either high likelihood or high severity—action required before deployment
- Medium: Moderate likelihood and severity—action required with defined timeline
- Low: Low likelihood and low severity—monitor and accept
Phase 5: Mitigation Measures
For each identified risk, document mitigation measures. Be specific about what actions are being taken, by whom, and by when.
Technical mitigations:
- Bias mitigation techniques applied during model development
- Output filters or constraints that prevent harmful outcomes
- Confidence thresholds that route uncertain decisions to human review
- Regular retraining and monitoring to address drift
Process mitigations:
- Human oversight requirements for high-stakes decisions
- Appeal and complaint mechanisms for affected individuals
- Regular bias audits and performance monitoring
- Incident response procedures for identified harms
Organizational mitigations:
- Training for system operators and decision-makers
- Clear policies on system use and limitations
- Accountability structures and escalation paths
- Ongoing stakeholder engagement
Phase 6: Monitoring Plan
The impact report is not a one-time exercise. Define how the system will be monitored on an ongoing basis.
- What metrics will be tracked and how frequently?
- Who is responsible for monitoring?
- What thresholds trigger action (e.g., if the selection rate ratio drops below 0.8)?
- How often will the full impact assessment be refreshed?
- How will changes to the system (model updates, data changes, use case expansion) trigger reassessment?
Phase 7: Public Summary
For systems where public disclosure is required or appropriate, create a public-facing summary of the impact report. This summary should be:
- Written in plain language accessible to non-technical audiences
- Honest about identified risks and bias findings
- Clear about mitigation measures and their effectiveness
- Specific about the system's purpose, scope, and limitations
- Inclusive of information about how affected individuals can appeal decisions or file complaints
Making Impact Reporting a Core Capability
For AI agencies, algorithmic impact reporting should be a service offering, not just a compliance exercise.
Build it into your project methodology. Include impact assessment as a standard phase in your AI development process. Assess impact during design, before deployment, and on an ongoing basis.
Develop templates and tools. Create standardized templates for impact reports that your team can adapt for each project. Build internal tools for bias analysis and risk assessment that make the process efficient and consistent.
Train your team. Every member of your AI team should understand what an impact assessment covers and why it matters. Developers should know how to collect the data needed for bias analysis. Project managers should know how to scope and schedule impact assessment work.
Stay current on requirements. The regulatory landscape for algorithmic impact assessment is changing rapidly. Assign someone in your organization to monitor regulatory developments and update your templates and processes accordingly.
Your Next Step
Select one AI system your agency has built or is building. Walk through the framework above—even informally—and assess its impact across the dimensions described. You do not need to produce a formal report to get value from the exercise. The act of systematically thinking through fairness, privacy, transparency, accountability, and safety will reveal issues you had not considered and will demonstrate to your team why impact assessment matters. Once you have done one informal assessment, formalize the process and build it into your standard project methodology.