An AI agency in Austin delivered a computer vision project three weeks late, twenty-two percent over budget, and with a model accuracy twelve points below what was promised in the SOW. The client accepted the deliverable but did not renew the contract. The agency lost a $340,000 annual relationship. When the founder asked the team what went wrong, she got three different answers from three different people, and none of them matched the project timeline in the PM tool.
Six months later, a different team at the same agency repeated almost the exact same mistakes on a similar project for a different client. Same underestimation of data cleaning effort. Same scope creep during model evaluation. Same communication gap between the ML engineer and the client's data team.
The problem was not that the agency failed once. Failure happens. The problem was that the agency had no mechanism to learn from failure. No post-mortem. No documented lessons. No process changes. Just a vague collective memory that "that project was rough" and an assumption that it would not happen again.
Post-mortems are how agencies turn expensive lessons into operational improvements. Here is how to run them well.
Why Most Agency Post-Mortems Fail Before They Start
The concept of a post-mortem is simple: after a project ends, you review what happened, identify what worked and what did not, and document actions to improve future delivery. But in practice, most agencies either skip post-mortems entirely or run them so poorly that they produce no value.
Common failure modes:
- They never happen. The project ends, the team moves to the next client, and nobody schedules the review. This is the most common failure. The agency is too busy delivering to stop and learn.
- They happen too late. Running a post-mortem two months after project completion means memories are fuzzy, context is lost, and the emotional urgency to improve has faded.
- They become blame sessions. Without clear facilitation, post-mortems devolve into finger-pointing. People get defensive, stop sharing honest assessments, and the exercise produces nothing useful.
- They produce documents nobody reads. The team generates a beautiful post-mortem report that gets filed in a shared drive and never influences a single future decision.
- They only happen after failures. If you only post-mortem disasters, you miss the chance to learn from your successes. Understanding why a project went well is just as valuable as understanding why one went badly.
Each of these failures is solvable with the right process.
When to Run a Post-Mortem
Run a post-mortem after every project. Not just the ones that went wrong. Every single one.
For ongoing retainer clients, run a post-mortem at natural checkpoints: quarterly reviews, major milestone completions, or when the scope shifts significantly.
Schedule the post-mortem within one week of project completion. Any longer and you lose the specificity that makes post-mortems valuable. People forget exact timelines, specific conversations, and the sequence of decisions that led to outcomes.
Block ninety minutes. Sixty minutes is too rushed for meaningful discussion. Two hours is too long for people to stay engaged. Ninety minutes gives you enough time to cover what happened, why it happened, and what to change, without dragging.
Who Should Be in the Room
The post-mortem needs everyone who was materially involved in the project. Not just the project manager. Not just the engineers.
Required attendees:
- Project manager or delivery lead. They own the timeline, budget, and scope narrative.
- Technical lead or senior engineer. They can speak to architecture decisions, technical challenges, and estimation accuracy.
- Individual contributors who did significant work. Their perspective on day-to-day friction is invaluable.
- Account manager or client-facing lead. They understand the client's experience and any communication gaps.
- A facilitator who was not on the project. This is critical. The facilitator keeps the discussion structured, prevents blame, and asks the probing questions that insiders might avoid.
Do not invite the client to the post-mortem. The internal post-mortem needs to be a safe space for honest assessment. If you want client feedback, gather it separately through a structured interview or survey, and bring that input into the post-mortem as data.
The Post-Mortem Framework That Produces Actionable Results
Structure matters. Without it, post-mortems wander, get stuck on one topic, or skip the most important issues. Use this framework to keep the conversation productive.
Phase One: Set the Context (15 minutes)
The facilitator opens by establishing the facts. This is not the time for opinions or interpretations. Just the data.
Present:
- Project name, client, and scope summary
- Original timeline versus actual timeline
- Original budget versus actual spend
- Key deliverables and their acceptance status
- Client satisfaction signals (NPS score, renewal decision, verbal feedback)
- Team satisfaction signals (if you surveyed the team)
State the ground rules:
- This is a blame-free zone. We are examining systems and processes, not judging individuals.
- Everyone's perspective is valuable. Junior team members are encouraged to speak up.
- Specific examples are more useful than general impressions.
- The goal is three to five concrete actions that will improve future projects.
Phase Two: What Went Well (20 minutes)
Start with successes. This is not just feel-good warm-up. Understanding what went well reveals processes worth replicating.
Prompt questions:
- What did we deliver that the client specifically praised?
- Where did our estimation prove accurate, and what made it accurate?
- Which communication practices kept the project on track?
- What technical decisions saved time or improved quality?
- Where did team collaboration work especially smoothly?
Document each success with enough specificity to be replicable. "Communication was good" is not useful. "Weekly fifteen-minute video calls with the client's data engineering lead prevented three potential integration issues" is useful.
Phase Three: What Did Not Go Well (30 minutes)
This is the core of the post-mortem. The facilitator's job here is to keep digging past surface-level symptoms to root causes.
Use the "Five Whys" technique. When someone identifies a problem, ask why it happened. Then ask why that happened. Keep going until you reach a systemic cause, not an individual error.
Example:
- The model accuracy was twelve points below the SOW target. Why?
- Because the training data had significant label noise that was not caught until week four. Why?
- Because we did not do a data quality audit during discovery. Why?
- Because our discovery process does not include a data quality checkpoint. Why?
- Because we built our discovery template for traditional software projects and never updated it for ML work.
Now you have something actionable: update the discovery template to include a data quality audit for any project involving model training.
Prompt questions for this phase:
- Where did scope change, and how was that change managed?
- Where did estimation miss, and by how much?
- What communication gaps existed between the team and the client?
- What technical risks materialized that were not anticipated?
- Where did dependencies or blockers cause delays?
- What tools or processes created friction instead of reducing it?
Phase Four: Root Cause Analysis (15 minutes)
Take the top three to five issues from Phase Three and dig into root causes using Five Whys or a similar technique. The facilitator should push past "someone made a mistake" to "our system allowed or encouraged this mistake."
Categories of root causes to look for:
- Process gaps: Missing checkpoints, unclear approval workflows, absent templates
- Estimation failures: Systematic underestimation of specific work types
- Communication breakdowns: Missing or misaligned stakeholders, unclear escalation paths
- Technical debt: Shortcuts that saved time in the short term but created problems later
- Resource mismatches: Wrong skills assigned to the project, or insufficient allocation
- Client management gaps: Unclear expectations, missing documentation, assumption mismatches
Phase Five: Action Items (10 minutes)
This is the most important phase. Without concrete actions, the post-mortem is just therapy.
Each action item must have:
- A specific, measurable change to a process, template, tool, or practice
- An owner who is responsible for implementing the change
- A deadline by which the change will be implemented
- A way to verify the change was implemented
Limit to three to five actions. More than that and nothing gets done. Pick the actions with the highest impact-to-effort ratio.
Good action item: "Add a data quality audit checklist to the discovery template. Owner: Sarah. Deadline: March 28. Verification: Template updated in Notion and reviewed by delivery lead."
Bad action item: "Be better at estimating." This is not specific, has no owner, no deadline, and no way to verify.
Documenting the Post-Mortem
The post-mortem document should be concise, scannable, and stored where future project teams will actually find it.
Template structure:
- Project summary: One paragraph covering scope, timeline, budget, and outcome
- Key metrics: Planned versus actual for timeline, budget, and quality targets
- What went well: Bulleted list with specific examples
- What did not go well: Bulleted list with root causes identified
- Action items: Table with description, owner, deadline, and status
- Open questions: Anything unresolved that needs further investigation
Store the document in your project management system, not a separate wiki. If your team uses Notion, put it in the project's Notion workspace. If you use Linear or Jira, link it to the project. The goal is that anyone starting a similar project in the future can find relevant post-mortems during their planning phase.
Building a Post-Mortem Culture
The process described above will produce value the first time you run it. But the real payoff comes from making post-mortems a cultural norm.
Make post-mortems mandatory, not optional. Add "post-mortem scheduled" as a required step in your project closure checklist. If the post-mortem does not happen, the project is not considered complete.
Celebrate post-mortem findings. When an action item from a post-mortem prevents a problem on a future project, highlight that win publicly. This reinforces the value of the process and encourages honest participation.
Track action item completion. If action items from post-mortems consistently go unimplemented, the team will stop taking the process seriously. Review outstanding post-mortem actions in your monthly operations meeting.
Share anonymized learnings across teams. If your agency has multiple delivery teams, create a quarterly digest of post-mortem insights. One team's hard-won lesson might save another team weeks of pain.
Never use post-mortem findings in performance reviews. The moment someone's honest assessment in a post-mortem shows up in their annual review, the entire system collapses. Post-mortems require psychological safety. Protect it absolutely.
Post-Mortem Patterns Specific to AI Agency Work
AI projects have recurring failure patterns that are worth watching for explicitly.
Data quality surprises. The client's data looked clean in the sample but was a mess at scale. Post-mortem action: require a full-data quality audit before committing to accuracy targets.
Evaluation metric mismatch. The team optimized for one metric while the client cared about a different one. Post-mortem action: document the specific evaluation metrics and their thresholds in the SOW, and get explicit client sign-off.
Infrastructure underestimation. The model worked in a notebook but deploying it to the client's environment took three times longer than expected. Post-mortem action: add an infrastructure assessment to the discovery phase and include deployment complexity in the estimate.
Scope creep through "just one more experiment." The client keeps asking for additional model variations, and the team keeps trying them without formal change orders. Post-mortem action: define the number of experiment iterations in the SOW and treat additional iterations as change orders.
Knowledge transfer failure. The model works but the client's team cannot maintain it. Post-mortem action: include documentation and training in the project scope and timeline, not as an afterthought.
Turning Post-Mortem Insights Into Process Improvements
The action items from individual post-mortems address specific issues. But over time, patterns emerge that call for larger process changes.
Review all post-mortem action items quarterly. Look for themes:
- If three post-mortems in a row cite estimation failures, your estimation process needs an overhaul, not a tweak.
- If multiple post-mortems mention communication gaps with the client's technical team, you need a standard stakeholder mapping step in your kickoff process.
- If data quality keeps surprising teams, your discovery framework is missing a critical component.
Assign an operations lead to own the post-mortem process. This person reviews all post-mortems, tracks action item completion, identifies patterns, and proposes systemic improvements. Without ownership, the process will atrophy.
Update your delivery playbooks based on post-mortem findings. Your playbooks should be living documents that evolve with every project. Post-mortems are the primary input for that evolution.
Your Next Step
If you have never run a post-mortem, start with your most recent completed project. Schedule ninety minutes with the project team within the next week. Use the five-phase framework from this post. Focus on producing three concrete action items with owners and deadlines.
If you already run post-mortems but they feel ineffective, audit your last five post-mortem documents. Check whether the action items were implemented. If fewer than half were completed, your problem is not the post-mortem process itself but the follow-through system. Assign an owner to track post-mortem actions and review them in your monthly operations meeting.
The agencies that grow sustainably are the ones that learn from every project. Post-mortems are the mechanism that makes that learning systematic rather than accidental. Start this week.