Most AI agencies finish a project, send the final invoice, and move on. The lessons from that engagement live in the heads of the people who worked on it. When those people get busy with the next project, the lessons evaporate.
A post-mortem changes that. It captures what worked, what did not, and what should change so that the agency actually gets better over time instead of just getting busier.
Why Post-Mortems Matter for AI Projects
AI projects have a higher learning surface than traditional software projects:
- model performance varies by use case in ways that are not always predictable
- data quality issues emerge during delivery, not during scoping
- client expectations around AI capabilities are frequently misaligned
- integration complexity often exceeds initial estimates
- production behavior differs from testing behavior in subtle ways
Each of these surprises contains a lesson. Without a post-mortem, each new project team rediscovers the same problems from scratch.
When to Run a Post-Mortem
Run a post-mortem within two weeks of project completion, while details are still fresh.
Also run post-mortems after:
- any project that significantly exceeded its timeline or budget
- any engagement where the client relationship was strained
- any incident that affected production systems or end users
- any project that was cancelled or paused before completion
The goal is not to post-mortem every project exhaustively. It is to capture high-value learnings consistently.
The Post-Mortem Framework
Part 1: Facts
Start with an objective summary of the engagement.
Document:
- project scope and original objectives
- timeline: planned versus actual
- budget: estimated versus actual
- team members involved
- key deliverables and their current status
- client satisfaction signal (explicit feedback, expansion, referral, etc.)
Facts should be stated without judgment. This section establishes the baseline for the discussion that follows.
Part 2: What Went Well
Identify the things that contributed to success.
Ask the team:
- What processes or practices worked particularly well?
- Where did the team exceed expectations?
- What tools or approaches saved time or improved quality?
- Where did communication with the client work smoothly?
- What would you definitely repeat on the next project?
Documenting successes is as important as documenting failures. Good practices need to be reinforced and shared, not just assumed.
Part 3: What Did Not Go Well
Identify the things that caused problems, delays, or quality issues.
Ask the team:
- Where did the project hit unexpected obstacles?
- What took longer than expected and why?
- Where did communication break down, internally or with the client?
- What assumptions turned out to be wrong?
- What risks were identified too late?
This section requires psychological safety. If team members feel blamed for problems, they will hide issues instead of surfacing them. Focus on systems and processes, not individuals.
Part 4: Root Causes
For each significant issue identified in Part 3, ask why it happened. Go at least two layers deep.
Example:
- Issue: Model accuracy was below the target threshold at launch.
- First why: The training data did not cover several important edge cases.
- Second why: Edge cases were not identified during discovery because the client's subject matter expert was not included in the data review.
- Root cause: The discovery process did not include a data validation step with the client's domain expert.
Root cause analysis turns symptoms into actionable improvements.
Part 5: Action Items
Convert root causes into specific, assignable actions.
Each action item should have:
- a clear description of what needs to change
- an owner who is responsible for implementing the change
- a deadline for completion
- a way to verify that the change was made
Examples of post-mortem action items:
- "Add a data validation step with client SME to the discovery checklist. Owner: Delivery Lead. Deadline: End of month."
- "Create a standard integration testing protocol for API-dependent projects. Owner: Engineering Lead. Deadline: Two weeks."
- "Update the scoping template to include a model performance baseline requirement. Owner: Solutions Architect. Deadline: Next sprint."
Action items without owners and deadlines do not get done. Be specific.
Running the Meeting
Duration: 60 to 90 minutes for most projects.
Attendees: Everyone who worked on the project. Optionally include the client for a joint review, which can strengthen the relationship.
Facilitation: Assign a facilitator who was not deeply involved in the project. This person keeps the discussion balanced and prevents it from becoming a blame session.
Format:
- Review the facts (10 minutes)
- Discuss what went well (15 minutes)
- Discuss what did not go well (20 minutes)
- Identify root causes for major issues (15 minutes)
- Define action items (15 minutes)
- Summary and next steps (5 minutes)
Documenting and Sharing
The post-mortem is only valuable if the insights are accessible to people who were not in the room.
Create a written summary that includes:
- project context and key metrics
- top three to five things that went well
- top three to five things that did not go well
- root causes and action items
Store post-mortem documents in a central location where team members can reference them when starting similar projects.
Review post-mortems quarterly to identify recurring themes. If the same issues appear across multiple projects, the root cause is systemic and requires a process-level change.
Common Post-Mortem Mistakes
Skipping the post-mortem because the project went well. Successful projects contain learnings too. Understanding why something worked is as valuable as understanding why something failed.
Turning the post-mortem into a blame session. This kills honesty and makes future post-mortems performative instead of productive.
Creating action items that nobody tracks. If the action items are not reviewed within thirty days, they were a waste of meeting time.
Only running post-mortems on failed projects. This creates a negative association with the process. Post-mortems should be a normal part of every significant engagement.
The Compound Effect
One post-mortem produces a few useful insights. Twenty post-mortems produce a knowledge base that fundamentally changes how the agency operates.
Over time, post-mortem-driven improvements compound:
- scoping accuracy increases because past estimation errors are documented
- delivery timelines shorten because common pitfalls are anticipated
- quality improves because testing protocols incorporate previous failures
- client satisfaction rises because the agency demonstrates continuous improvement
The agencies that get better every quarter are not working harder. They are learning systematically. Post-mortems are the mechanism that makes that learning happen.