A playbook is not a tutorial. A tutorial teaches you a technique once. A playbook tells you which move to make, when to make it, and who owns the outcome, so the same project does not stall in the same place every time. Object detection projects fail less because the math is hard and more because nobody decided who handles labeling disputes or what triggers a retrain.
This is an operating playbook for understanding how AI detects objects in images and turning that understanding into a shipped system. Each play has a trigger (the situation that calls for it), an owner (the role accountable), and a sequence (the steps in order). Run them roughly in the order presented, but treat them as plays you call rather than chapters you read.
If your team has ever watched a promising vision prototype rot in a notebook for three months, this is for you.
Play 1: Frame The Decision The Model Serves
Trigger: Someone proposes "let's use object detection for X." Owner: Product lead.
Before any model, write one sentence: when the model detects this object, the system does that. Detection is never the goal; it is an input to a decision. Counting parts on a conveyor, blurring faces, flagging empty shelves, each implies a different accuracy profile and failure cost.
Sequence
- Name the downstream action the detection triggers.
- State the cost of a false positive versus a false negative in plain terms.
- Decide whether errors are recoverable (a human reviews) or not (the system acts automatically).
This single page determines every later tradeoff. If you cannot write it, you are not ready to build.
Play 2: Decide Build, Fine-Tune, Or Buy
Trigger: The decision is framed and you need a model. Owner: Engineering lead.
Most teams reach for training when they should reach for an API, or reach for an API when their objects are too specialized for one. The default should be fine-tuning a pretrained model; from-scratch training and generic APIs are the exceptions.
Sequence
- Check whether a hosted detection API already recognizes your object categories well enough.
- If not, identify a pretrained model close to your domain to fine-tune.
- Reserve from-scratch training for genuinely novel object types with abundant data.
The framework for choosing an approach walks through this decision tree in more depth.
Play 3: Run A Throwaway Spike
Trigger: You have picked an approach but not committed budget. Owner: A single engineer, time-boxed to a few days.
Take a pretrained model, point it at fifty of your real images, and look at the raw output. This is the cheapest information you will ever buy. You learn immediately whether the problem is "mostly works, needs tuning" or "fundamentally hard for our data."
Sequence
- Collect fifty representative images from production, including the ugly ones.
- Run an off-the-shelf detector and eyeball every result.
- Categorize failures: missed small objects, wrong labels, bad boxes, or unseen categories.
Do not polish anything. The spike exists to kill bad ideas before they consume a quarter.
Play 4: Build The Labeling Pipeline
Trigger: The spike shows fine-tuning is needed. Owner: Data lead.
Labeling is where projects quietly die. Inconsistent boxes teach the model contradictions, and ambiguous guidelines produce inconsistent boxes. Treat your labeling guide as a real document with examples of correct and incorrect annotations.
Sequence
- Write a labeling guide with edge cases: occlusion rules, minimum object size, what counts as the object boundary.
- Label a small batch, then have a second person relabel it and measure disagreement.
- Resolve every disagreement by updating the guide, not by arguing per image.
A repeatable workflow for object detection makes this pipeline reusable across projects instead of rebuilt each time.
Play 5: Establish A Real Evaluation Set
Trigger: Labeling has begun. Owner: Data lead, reviewed by engineering.
Carve out a held-out test set that mirrors production reality and never touch it during training. This set is your source of truth. If it does not look like the field, your metrics will lie to you and you will not find out until users do.
Sequence
- Sample test images across lighting, angles, object sizes, and rare categories.
- Lock the set; no peeking, no training on it, ever.
- Define the metric that matches your decision, not just generic mean average precision.
Play 6: Fine-Tune And Iterate On Errors
Trigger: Labeled training data and a frozen test set exist. Owner: Engineering lead.
Fine-tune, evaluate, then study the worst failures rather than the aggregate score. The aggregate hides where the model is dangerously wrong. Error-driven iteration beats blind hyperparameter sweeps almost every time.
Sequence
- Fine-tune the pretrained model on your labeled data.
- Pull the lowest-confidence correct predictions and the highest-confidence mistakes.
- Add targeted examples for the failure patterns and retrain.
Beware tuning your suppression and confidence thresholds on the test set; that quietly corrupts your evaluation, a trap detailed in the best practices guide.
Play 7: Define The Confidence Policy
Trigger: The model meets the metric on the test set. Owner: Product lead with engineering.
The model emits confidence scores; you decide what to do with each band. This is a policy, not a default. A high-stakes automated action needs a high threshold and possibly a human checkpoint; a low-stakes suggestion can tolerate noise.
Sequence
- Set the confidence threshold from the false-positive and false-negative costs in Play 1.
- Decide what happens to medium-confidence detections: route to human, drop, or act.
- Document the policy so it survives the engineer who wrote it.
Play 8: Ship With A Drift Tripwire
Trigger: The model is ready for production. Owner: Engineering lead, on a recurring schedule.
Models degrade when the world changes: new product packaging, a moved camera, a different season. Deploy with monitoring that catches drift before customers do, and define what triggers a retrain.
Sequence
- Log a sample of production predictions for periodic human spot-checks.
- Set a tripwire: if the spot-check accuracy drops below a line, a retrain is queued.
- Assign the owner who responds when the tripwire fires.
For a concrete account of these plays surviving contact with reality, see Case Study: How Ai Detects Objects in Images in Practice.
Frequently Asked Questions
What if I only have a tiny budget and a few days?
Run Play 1 and Play 3 only. Frame the decision in one sentence, then spike an off-the-shelf detector against your real images. Those two plays cost almost nothing and tell you whether the full sequence is worth funding. Most doomed projects get killed right here, which is exactly the point.
Who should own the labeling guide?
A dedicated data lead, not the engineer training the model. When the same person labels and trains, blind spots in the guidelines never surface because the trainer unconsciously compensates. Separating the roles forces the ambiguities into the open where they can be resolved consistently.
How often should we retrain?
There is no fixed cadence; retrain on a trigger, not a calendar. The trigger is your drift tripwire from Play 8 firing, or a known change in the environment such as new packaging or relocated cameras. Calendar-based retraining wastes effort when nothing changed and lags reality when something did.
Can we skip the held-out test set if we are in a hurry?
No. The test set is the one shortcut that always backfires. Without it, you have no honest measure of whether the model works, and you will discover failures in production where they cost the most. Skipping any play before this one is recoverable; skipping this one is not.
How do I know which metric to optimize?
Derive it from Play 1. If missing an object is catastrophic, optimize recall. If false alarms overwhelm reviewers, optimize precision. Generic mean average precision is a fine research metric but a poor business one, because it averages away the specific error your application cannot tolerate.
Key Takeaways
- Every play has a trigger and an owner; ambiguity about who acts is what stalls real projects.
- Frame the downstream decision in one sentence before touching a model, and derive your metric from it.
- Default to fine-tuning a pretrained model; spike against real images before committing budget.
- Labeling consistency and a locked held-out test set are non-negotiable foundations, not formalities.
- Ship with a drift tripwire and a named owner, because models decay when the world moves and yours will.