Establishing an Ethical Review Board for AI Projects: A Step-by-Step Agency Guide

A 30-person AI agency was asked to build a predictive policing tool for a mid-sized city's police department. The project was lucrative — six figures over 18 months — and technically straightforward. But when the lead data scientist raised concerns about reinforcing racial bias in policing, there was no formal process for evaluating those concerns. The agency's founder overruled the objection because the revenue was needed. The project launched, drew immediate criticism from civil rights organizations, and became a PR disaster that cost the agency three other clients who didn't want to be associated with predictive policing. Six months later, the city council terminated the contract anyway.

If that agency had an ethical review board, the outcome could have been different. Not necessarily a different decision — maybe they still would have taken the project — but a better-informed decision with documented reasoning, mitigation plans, and contingency strategies. Instead, one person made a gut call, and the agency paid the price.

Ethical review boards aren't just for Google and Microsoft. Any agency building AI systems that affect people needs a structured process for evaluating the ethical implications of their work. This guide shows you how to build one that's right-sized for an agency, practical enough to actually use, and rigorous enough to make a difference.

Why Agencies Need Ethical Review

The case for ethical review at agencies is both principled and pragmatic.

The principled case: AI systems can cause real harm to real people. When your agency builds a model that determines who gets a loan, who gets hired, what content people see, or how resources are allocated, you have a responsibility to consider the implications. Ethical review provides a structured way to fulfill that responsibility.

The pragmatic case: Ethical review protects your business. It prevents you from taking projects that will blow up in your face. It helps you identify and mitigate risks that would otherwise surface after delivery. It differentiates you from competitors who don't take ethics seriously. And it builds trust with enterprise clients who increasingly require evidence of ethical governance in their vendor evaluations.

The regulatory case: As AI regulations proliferate, many require some form of ethical assessment before deployment. The EU AI Act requires conformity assessments for high-risk systems. Various US state laws require impact assessments. Having an ethical review board positions your agency to meet these requirements systematically.

The talent case: AI professionals increasingly want to work for organizations that take ethics seriously. An ethical review board signals to prospective hires that your agency cares about the impact of its work, which helps you attract and retain top talent.

Designing Your Ethical Review Board

Determine the Board's Scope

Before you recruit members or draft a charter, decide what the board will and won't do.

In scope:

• Reviewing new project proposals for ethical concerns before the agency commits
• Evaluating ethical issues that arise during active projects
• Establishing ethical guidelines and policies for the agency
• Advising on client-facing ethical questions (e.g., "Should we build this?")
• Providing guidance on fairness, bias, transparency, and accountability

Typically out of scope:

• Day-to-day project management decisions
• Technical architecture choices (unless they have ethical implications)
• Business strategy and pricing
• Individual personnel decisions

The board should be advisory, not executive. It makes recommendations; the agency's leadership makes final decisions. This structure ensures the board is consulted but doesn't create a bottleneck in project delivery.

Define Review Triggers

Not every project needs a full ethical review. Define clear criteria for when review is triggered.

Mandatory review triggers:

• The AI system makes or influences decisions about individuals (hiring, lending, insurance, criminal justice, healthcare)
• The project involves sensitive data categories (health data, financial data, biometric data, children's data)
• The AI system could cause physical safety risks
• The project involves law enforcement, military, or intelligence applications
• The client or project has been the subject of ethical controversy

Discretionary review triggers:

• A team member raises an ethical concern about a project
• The project involves a novel application of AI that hasn't been evaluated before
• The project could have significant environmental impact
• The client's industry has a history of ethical concerns related to AI

Exempt from review:

• Internal tooling projects that don't affect external stakeholders
• Standard analytics and reporting projects
• Projects that have been reviewed previously and haven't materially changed

Select Board Members

The composition of your ethical review board determines its effectiveness. You need diverse perspectives, relevant expertise, and practical experience.

Internal members (2-3 people):

• A senior technical leader who understands the capabilities and limitations of AI systems
• A business leader who understands the agency's strategic context, client relationships, and financial constraints
• A project delivery lead who understands the practical realities of building and deploying AI systems

External members (2-3 people):

• An ethicist or social scientist with expertise in technology ethics, preferably with experience in the industries your agency serves
• A legal expert with knowledge of AI regulation, data protection law, and liability
• A domain expert or community representative from a population likely to be affected by your agency's work

External members are essential. Internal-only boards tend to develop blind spots and groupthink. External members bring perspectives that your team may not have considered and provide credibility that internal-only boards lack.

Compensation: Pay your external board members for their time. This ensures you attract qualified individuals and signals that you value their contribution. A quarterly retainer plus per-review fees is a common structure for agency-sized boards.

Terms: Set term limits (2-3 years) with staggered rotation to maintain continuity while bringing in fresh perspectives.

Draft the Board Charter

Your charter is the governing document for the ethical review board. It should cover:

• Mission statement — Why the board exists and what it aims to achieve
• Scope of authority — What the board reviews, what it doesn't, and whether its recommendations are binding or advisory
• Membership — How members are selected, their terms, and how vacancies are filled
• Meeting frequency — How often the board convenes for regular meetings (quarterly is typical) and the process for ad hoc reviews
• Review process — How projects are submitted for review, what information must be provided, and the timeline for decisions
• Decision-making — How the board reaches recommendations (consensus, majority vote, etc.)
• Confidentiality — How confidential client and project information is protected
• Reporting — How the board's activities and recommendations are reported to agency leadership
• Accountability — How the board's effectiveness is evaluated and how its recommendations are tracked

Keep the charter concise and practical. A 30-page charter that nobody reads is worse than a 3-page charter that everyone follows.

The Review Process

Step 1: Project Submission

When a project triggers a review, the project lead submits a review request that includes:

• Project description — What the AI system will do, who the client is, and who will be affected
• Ethical risk assessment — The project lead's initial assessment of the ethical risks, using a standardized template
• Technical approach — A summary of the data sources, model type, and deployment plan
• Mitigation measures — Any steps already planned to address ethical risks
• Open questions — Specific questions the project lead wants the board to address

The submission should be concise — 2-4 pages maximum. The board needs enough information to evaluate the ethical implications without being overwhelmed by technical detail.

Step 2: Board Review

The board reviews the submission and may request additional information or a presentation from the project team. For straightforward reviews, the board can evaluate the submission asynchronously. For complex or high-risk projects, schedule a meeting where the project team presents and answers questions.

During the review, the board should consider:

• Who could be harmed by this AI system, and how?
• Are the proposed mitigation measures adequate?
• Are there populations or use cases that haven't been considered?
• Does the agency have the expertise to build this system responsibly?
• What would the consequences be if the system failed or was misused?
• Are there ethical red lines that this project crosses or approaches?

Step 3: Board Recommendation

The board issues a recommendation that falls into one of four categories:

• Approved — The project can proceed as proposed
• Approved with conditions — The project can proceed if specific conditions are met (e.g., additional fairness testing, enhanced monitoring, client disclosures)
• Deferred — The board needs more information or the project needs to be redesigned before it can be evaluated
• Not recommended — The board recommends against taking the project

For each recommendation, the board should provide a written rationale that explains the reasoning and any conditions. This documentation is valuable for institutional learning and for demonstrating due diligence.

Step 4: Leadership Decision

Agency leadership reviews the board's recommendation and makes a final decision. If leadership decides to proceed with a project the board recommended against, the rationale should be documented. This creates accountability without making the board's recommendations binding in a way that could paralyze decision-making.

Step 5: Follow-Up

For projects that proceed with conditions, the board should follow up to verify that the conditions are being met. This can be done through periodic check-ins with the project team or by reviewing key deliverables (e.g., fairness assessment results) before they're finalized.

Making It Work in Practice

Keep It Lightweight

The biggest risk to an ethical review board is that it becomes burdensome enough to be circumvented. Design the process for speed and simplicity.

• Set a target turnaround time for reviews (one week for standard reviews, 48 hours for urgent reviews)
• Use standardized templates and checklists to reduce the effort required for submissions and reviews
• Allow asynchronous review for straightforward projects
• Reserve in-person meetings for genuinely complex or controversial cases

Build a Precedent Library

Over time, your board will review similar types of projects. Document the board's reasoning for each review and build a searchable library of precedents. This serves several purposes.

• Efficiency — When a similar project comes up, the board can reference past precedents rather than starting from scratch
• Consistency — Precedents help the board make consistent recommendations across similar cases
• Training — New board members can review precedents to understand the board's approach and values
• Accountability — A record of past decisions demonstrates the board's track record and reasoning

Integrate with Project Lifecycle

Don't make ethical review a separate, disconnected process. Integrate it with your existing project lifecycle.

• Add an ethical review checkpoint to your project scoping process
• Include ethical review status in your project management dashboards
• Reference ethical review recommendations in project kickoff meetings
• Follow up on ethical review conditions during sprint reviews or milestone checkpoints

Handle Disagreements Constructively

The board will sometimes disagree with project teams or with agency leadership. This is healthy — it means the board is doing its job. Handle disagreements constructively.

• Give project teams the opportunity to respond to the board's concerns and propose alternative approaches
• If leadership overrides the board, document the reasons and revisit the decision after the project is underway to evaluate whether the board's concerns were valid
• Never penalize team members for raising ethical concerns, even if the board ultimately clears the project

Measure Effectiveness

Track metrics that indicate whether the board is adding value.

• Review volume — How many projects are being reviewed? If the number is very low, the review triggers may need to be broadened.
• Turnaround time — How long does the review process take? If it's consistently too slow, the process needs streamlining.
• Outcome distribution — What proportion of projects are approved, approved with conditions, deferred, or not recommended? A board that approves everything isn't adding value; a board that blocks everything isn't practical.
• Condition compliance — When projects are approved with conditions, are the conditions being met?
• Post-deployment outcomes — For reviewed projects, how often do ethical issues arise after deployment? A well-functioning board should reduce post-deployment ethical incidents.
• Team satisfaction — Do project teams find the review process valuable or burdensome? Regular feedback helps you calibrate the process.

Communicating Your Ethical Review Board to Clients

Your ethical review board is a selling point, not something to hide. Here is how to communicate it effectively.

In proposals and pitches, mention your ethical review process as part of your governance framework. Explain that projects involving sensitive applications undergo ethical review before and during development. This demonstrates maturity and responsibility.

During project kickoffs, walk the client through the ethical review process. Explain what was reviewed, what concerns were identified, and what conditions apply. This builds trust and sets expectations.

In case studies and marketing, reference your ethical review process (without disclosing confidential details). Position it as a competitive advantage that sets your agency apart from firms that build AI without ethical oversight.

When challenges arise, reference the ethical review process to demonstrate that you anticipated and prepared for the challenge. "Our ethical review board identified this risk during the scoping phase, and we implemented these mitigation measures" is a much stronger response than "We didn't foresee this issue."

Scaling the Board as Your Agency Grows

As your agency grows, your ethical review board needs to evolve.

At 10-30 employees, a single board with 4-5 members reviewing all triggered projects is sufficient. The founder or CEO should be involved in the process to signal its importance.

At 30-100 employees, consider adding domain-specific sub-committees. If your agency works in healthcare and financial services, you might have a healthcare ethics sub-committee and a financial services ethics sub-committee, each with relevant domain expertise.

At 100+ employees, formalize the board into a standing committee with dedicated staff support. Create a full-time ethics lead role that manages the review process, maintains the precedent library, and coordinates between the board and project teams.

Regardless of size, maintain external representation on the board. As your agency grows, the temptation to internalize the board increases. Resist it. External perspectives become more important, not less, as the agency takes on larger and more complex projects.

Your Next Steps

This week: Identify the three most ethically sensitive projects your agency has delivered in the past year. For each one, ask: Would an ethical review board have changed the approach? What concerns would the board have raised? How would the outcome have been different?

This month: Draft a board charter and identify potential internal and external members. Socialize the concept with your leadership team and get buy-in.

This quarter: Launch the board with a pilot period. Review 3-5 projects, refine the process based on feedback, and formalize the board as a permanent part of your governance framework.

An ethical review board isn't a luxury or a PR stunt. It's a practical governance mechanism that helps your agency make better decisions about the work you take on and how you deliver it. The agency in the opening scenario learned that lesson the hard way. You don't have to.