A playbook is different from a guide. A guide explains concepts; a playbook tells you what to do, when to do it, and who's responsible. This is the operating playbook for the zero-shot versus few-shot decision, written so a team can run it without re-deriving the logic every time. Each play has a trigger that fires it, an owner who runs it, and a place in the sequence.
The reason to operationalize this at all is that the decision recurs constantly, across every new task, every model upgrade, every drift event, and re-litigating it from scratch each time wastes effort and produces inconsistent results. A playbook turns a judgment call into a repeatable procedure with defined hand-offs. Run the plays in order; escalate when a trigger fires.
This pairs naturally with A Framework for Zero Shot vs Few Shot Learning, which supplies the decision logic the plays execute. Here we focus on sequencing and ownership.
Play 1: Baseline First
Trigger: Any new task enters the pipeline.
Owner: The person building the prompt.
Procedure: Write a clear zero-shot instruction, specifying format and edge cases, and run it on at least twenty representative inputs including hard cases. Record the error rate and the kinds of errors. Do not add examples yet.
This play is non-negotiable and comes first every time. The baseline is the reference point every later decision is measured against. Skipping it is how teams end up paying the example tax on tasks that never needed it. Getting Started with Zero Shot vs Few Shot Learning details the mechanics.
Play 2: Decide Whether Examples Are Even Warranted
Trigger: Baseline error rate is known.
Owner: Prompt builder, with a reviewer for high-stakes tasks.
Procedure: Apply the decision rule. If the baseline error rate is acceptable for the use case and errors are cheap, stop here and ship zero-shot. If the baseline shows a consistent, fixable error class, or the task is regulated or hard to describe, proceed to Play 3.
The key discipline is that examples must be justified by the baseline, not added reflexively. Most general, high-volume tasks exit here with a clean zero-shot prompt.
Play 3: Build and Test the Few-Shot Variant
Trigger: Play 2 determined examples are warranted.
Owner: Prompt builder.
Procedure: Select two to three examples that target the specific errors the baseline made. Mirror the real input distribution, including hard cases, and balance labels. Run the identical test set and compare error rates against the baseline.
- If few-shot clearly beats the baseline and the token cost is justified, proceed to Play 4.
- If it doesn't, return to Play 2; the issue may be the instruction, not the absence of examples.
Change one variable at a time so you can attribute the improvement. A Step-by-Step Approach to Zero Shot vs Few Shot Learning covers the test mechanics.
Play 4: Cost the Decision
Trigger: A few-shot variant outperforms the baseline.
Owner: Prompt builder, escalating to a budget owner at high volume.
Procedure: Calculate the per-call token overhead times expected volume, and weigh it against the errors prevented times the cost per error. Confirm few-shot wins on total cost, not just accuracy. At high volume, surface this to whoever owns the token budget.
This play prevents accuracy tunnel vision. A few-shot prompt that's more accurate but costs more in tokens than the errors it prevents is a bad trade. The ROI of Zero Shot vs Few Shot Learning gives the calculation.
Play 5: Govern Sensitive Workflows
Trigger: The prompt touches personal data, money, or regulated output.
Owner: A reviewer with governance responsibility.
Procedure: Confirm few-shot examples contain no real customer data, that labels are balanced to avoid bias, and that outputs are audited by segment, not just in aggregate. Register the prompt with an owner and a review date.
This play exists because few-shot's hidden risks, data leakage in examples, baked-in bias, fluent-but-wrong output, are exactly the ones that hurt on sensitive workflows. The Hidden Risks of Zero Shot vs Few Shot Learning is the reference.
Play 6: Register and Document
Trigger: A prompt is approved to ship.
Owner: Prompt builder.
Procedure: Add the prompt to the shared registry with its baseline, its chosen approach, the reasoning, the example set (if any), the measured error rate, and a named owner. Version the example set like code.
Documentation is what keeps a prompt from becoming untouchable folklore. It makes the prompt debuggable by the next person and recoverable when a model changes.
Play 7: Re-Measure on Drift or Model Change
Trigger: Inputs drift, volume changes by more than 2x, or the model is upgraded.
Owner: The registered prompt owner.
Procedure: Re-run the baseline and, if applicable, the few-shot variant against fresh, labeled data. Refresh stale examples. Re-cost at the new volume. Update the registry.
This play closes the loop. The right choice moves over time, and without a scheduled re-measurement, few-shot accuracy degrades silently and cost decisions go stale. This is the play teams most often skip and most often regret.
Escalation Rules and Stop Conditions
A playbook needs to tell people not just what to run but when to stop and when to pull someone else in. Three escalation rules keep the plays from running off the rails.
- Stop at Play 2 when the baseline is good enough. If the zero-shot baseline meets the bar and errors are cheap, the correct move is to ship and stop. Continuing into few-shot "just to be safe" is how teams accumulate unnecessary token cost. A clean exit is a success, not a shortcut.
- Escalate to a budget owner at Play 4 when volume is high. Below a modest volume threshold the token overhead is a rounding error and the builder decides alone. Above it, the few-shot example tax becomes a real line item and someone who owns the budget should sign off.
- Escalate to governance at Play 5 on sensitive data. The moment a prompt touches personal data, money, or regulated output, it leaves the builder's sole authority. This is a hard stop, not a judgment call.
These rules exist because the most expensive mistakes happen when people either over-engineer a forgiving task or quietly ship a risky one without review. Encoding the escalations into the playbook removes the ambiguity about who decides.
A Worked Example of the Sequence
Consider a support-ticket classifier. Play 1 produces a zero-shot baseline at a 9% error rate, with most errors confusing refund requests for technical issues. Play 2 sees a consistent, fixable error class and proceeds. Play 3 adds three balanced examples, including two refund cases, and the error rate drops to 3%. Play 4 confirms the example overhead is trivial at this volume and the prevented errors save real review time, so few-shot wins. Play 5 doesn't fire because no sensitive data is involved. Play 6 registers the prompt with its baseline, examples, and an owner. Three months later, Play 7 fires on a model upgrade, the team re-runs the baseline, finds the new model hits 4% zero-shot unaided, and retires the examples to save tokens. That full arc is the playbook working as designed.
Running the Plays as a Sequence
The default flow is linear: Play 1 to 2, then either ship zero-shot or continue 3 through 6, with Play 7 firing on its own triggers forever after. The escalation points are Play 4 (to a budget owner at high volume) and Play 5 (to governance on sensitive data). Everything else lives with the prompt builder. Adapt the owners to your org, but keep the sequence and the triggers intact, because the order is what prevents the two classic failures: shipping few-shot without justification, and never re-measuring.
Frequently Asked Questions
Why does the playbook always start with a zero-shot baseline?
Because the baseline is the reference every later decision depends on. Without it, you can't tell whether examples actually help, can't justify their token cost, and can't detect later degradation. Starting with the baseline is what keeps the rest of the plays grounded in evidence.
Who owns the decision to use few-shot?
The prompt builder owns the default decision, with a reviewer involved for high-stakes or regulated tasks and a budget owner escalated to at high volume. Ownership is split by stakes: low-stakes prompts stay with the builder, while sensitive or expensive ones pull in governance and budget owners.
What triggers a re-measurement of an existing prompt?
Three triggers: input data drifting, volume changing by more than roughly 2x, or a model upgrade. Any of these can shift the right choice, so the registered prompt owner re-runs the baseline and few-shot variant against fresh data. Skipping this is the most common reason prompts silently degrade.
How is this different from just following a framework?
A framework gives you the decision logic; the playbook adds triggers, owners, and sequencing so a team executes it consistently. The framework tells you what's right for one task; the playbook makes sure every task, every drift event, and every model change runs the same procedure with clear hand-offs.
Can a small team skip some plays?
A small team can compress ownership, with one person running several plays, but shouldn't skip the plays themselves. The baseline, the justification, the cost check, and the re-measurement each prevent a specific failure. What scales down is the number of people involved, not the number of checkpoints.
Key Takeaways
- The playbook turns a recurring judgment call into a repeatable procedure with triggers, owners, and a fixed sequence.
- Always start with a zero-shot baseline; examples must be justified against it, never added reflexively.
- Cost the decision and govern sensitive workflows as explicit plays, not afterthoughts.
- Register and document every shipped prompt so it stays debuggable and recoverable.
- Re-measure on drift, volume change, or model upgrade; this is the most-skipped, most-regretted play.