Decomposition adds cost and complexity, so the only way to know it was worth it is to measure. Yet most teams that build multi-step pipelines never instrument them properly. They feel that the pipeline is better, ship it, and never check whether the added steps actually moved any number that matters.
This piece defines the metrics worth tracking for decomposition prompting, explains how to instrument each one, and shows how to read the signal so you can act on it. The metrics fall into three groups: quality metrics that tell you if the output is good, efficiency metrics that tell you what it costs, and pipeline-health metrics that tell you where failures originate.
The unifying principle is comparison. Almost every metric is most useful when measured against the single-prompt baseline, because that comparison is what proves decomposition earned its place.
Quality Metrics
First-pass acceptance rate
The share of outputs that pass review without needing rework. This is the headline quality metric because it captures whether the pipeline produces usable results. Instrument it by logging each output's review outcome and tracking the percentage accepted over time.
Error rate by category
Not all errors are equal. Track distinct error types, such as factual mistakes, contradictions, and truncations, separately. Categorized errors tell you which reasoning phase is failing, which is far more actionable than a single error number. The categories map directly to the failure types in our common mistakes guide.
Consistency across runs
Run the same input several times and measure how much the output varies. High variance signals a missing shared contract or an under-specified step. This metric is especially valuable for pipelines that must produce uniform output, like the competitor profiles in our examples piece.
Efficiency Metrics
Cost per acceptable output
Total token cost divided by the number of outputs that pass review. This is more honest than raw token cost because it accounts for rework. A cheap pipeline that produces many unusable outputs may cost more per acceptable result than an expensive one that rarely fails.
Latency per output
End-to-end time from input to final output. Decomposition adds latency with each step, so track this to know whether your pipeline is fast enough for its use case. Per-step latency, if your tooling exposes it, tells you which step is the bottleneck.
Token spend by step
Breaking token cost down by step reveals which steps are expensive. A step that costs a lot and contributes little to quality is a candidate for removal, a pruning decision the trade-offs piece frames in detail.
Pipeline-Health Metrics
Step-level failure attribution
When the final output is wrong, you want to know which step caused it. Instrument validation at boundaries and log which checks fail. Over time, this builds a map of where your pipeline is fragile.
Handoff integrity
Track how often a handoff arrives malformed or missing required fields. Frequent handoff failures point to an upstream step that does not reliably produce its contract, which is often the real root cause of downstream errors.
Recombination conflict rate
Measure how often the recombination step has to resolve contradictions between subtask outputs. A high rate suggests your steps are producing inconsistent results that the merge is papering over, which is a sign the decomposition itself needs rework.
How to Read the Signals Together
The baseline comparison is the point
Every metric should be read against the single-prompt baseline. If first-pass acceptance is no higher than the baseline, the pipeline is not earning its cost. If cost per acceptable output is worse than the baseline, you have made things worse.
Diagnose before you optimize
Use error categories and step-level attribution to find where the problem lives before changing anything. Optimizing a step that is not the bottleneck wastes effort. The metrics exist to point you at the right step, which connects to the diagnostic spirit of our framework.
Watch for drift over time
Pipelines degrade as models change and inputs shift. Track your headline metrics continuously so you catch degradation early rather than discovering it through a client complaint.
Instrumenting Metrics Without Drowning in Data
Start with one metric per group
You do not need every metric from day one. Pick the headline from each group: first-pass acceptance for quality, cost per acceptable output for efficiency, and step-level failure attribution for health. Three numbers, tracked honestly against the baseline, tell you more than a dashboard of twenty metrics nobody reads. Add deeper metrics only when a headline number signals a problem worth diagnosing.
Log at the boundaries, not just the ends
The temptation is to log only the final output and its review outcome. That tells you whether the pipeline works but not why it fails. Logging the structured handoff at each boundary turns a black box into something you can debug. When the final output is wrong, the boundary logs let you walk backward to the step that first violated its contract.
Make review outcomes structured
If your quality metric depends on human review, capture the review outcome in a structured form: accepted, minor rework, major rework, plus the error category. Free-text review notes are hard to aggregate. Structured outcomes let you compute acceptance rates and error distributions automatically, which is what makes continuous tracking sustainable rather than a manual chore.
Frequently Asked Questions
What is the single most important metric for a decomposition pipeline?
First-pass acceptance rate measured against the single-prompt baseline. It directly answers whether the pipeline produces usable output and whether decomposition improved on the simpler alternative. Cost and health metrics matter, but if first-pass acceptance is not better than the baseline, the pipeline has not justified its added complexity regardless of other numbers.
Why measure cost per acceptable output instead of raw token cost?
Because raw token cost ignores rework. A pipeline with low token cost that frequently produces unusable output may require so much human rework that its true cost is higher than a pricier pipeline that rarely fails. Cost per acceptable output captures the full economic picture, including the human time spent fixing failures.
How do I attribute a final-output error to a specific step?
Instrument validation at boundaries and log which checks fail. When the final output is wrong, the boundary logs tell you which step's output first violated its contract. Without boundary instrumentation, you are left guessing, and tracing an error back through several steps by hand is slow and unreliable.
What does high consistency variance tell me?
It usually signals a missing shared contract or an under-specified step. If the same input produces noticeably different outputs across runs, some step lacks the constraints needed to produce stable results. High variance is especially damaging for pipelines that must produce uniform output, where it directly undermines the value of the work.
How often should I re-check these metrics?
Headline quality and cost metrics deserve continuous tracking so you catch drift early. Deeper diagnostic metrics like step-level attribution can be reviewed when a problem appears or on a regular cadence. The key is not to instrument once and forget; pipelines degrade as models and inputs change, and only ongoing measurement reveals it.
Can I justify a pipeline that costs more but is more reliable?
Yes, when reliability matters more than cost for the task. For client-facing or high-stakes work, a more expensive but more reliable pipeline is easily justified because errors carry reputational or financial consequences far larger than token cost. The metrics let you make that trade explicitly rather than assuming one side wins.
Turning Metrics Into Action
Set thresholds, not just dashboards
A metric you only watch is half a metric. Decide in advance what value triggers an action: a first-pass acceptance rate below a threshold means a step needs rework, a cost per acceptable output above a threshold means a step needs pruning. Thresholds convert passive monitoring into a decision system, so a degrading number prompts a response instead of sitting unnoticed on a dashboard nobody acts on.
Tie each metric to an owner
Metrics without owners drift. Assign each headline metric to a person responsible for watching it and acting when it crosses a threshold. The owner does not have to fix the problem alone, but they are accountable for noticing it and raising it. Unowned metrics are the ones that quietly degrade until a client, rather than a teammate, reports the failure.
Close the loop after every change
When you change a step in response to a metric, measure again to confirm the change helped. It is easy to assume a fix worked and move on, but pipelines are full of interactions where fixing one step shifts the failure to another. Re-measuring after each change against the baseline is what keeps the improvement real rather than imagined.
Key Takeaways
- Track quality metrics like first-pass acceptance, categorized error rate, and cross-run consistency.
- Track efficiency metrics like cost per acceptable output, latency, and token spend by step.
- Track pipeline-health metrics like step-level failure attribution, handoff integrity, and recombination conflict rate.
- Read every metric against the single-prompt baseline, because that comparison proves decomposition earned its cost.
- Diagnose with error categories and step attribution before optimizing, and monitor continuously to catch drift.