Once you have a flat schema returning clean output, structured generation feels solved. Then the real requirements arrive: deeply nested objects, fields whose type depends on another field, arrays of variable length, output that has to stream token by token into a UI, and recovery from a response that is ninety percent correct. The patterns that got you to your first result do not scale to these, and the failure modes get subtle.
This piece is for practitioners who already know the fundamentals. It covers the edge cases that separate a working prototype from a system that holds up under real input diversity, and the expert nuances that are easy to get wrong even when you know they exist.
We will work through schema complexity, union and conditional types, streaming, partial recovery, and the discipline of evaluating structured output rigorously.
Taming Schema Complexity
Depth Costs Reliability
Every level of nesting and every additional field gives the model another place to drift. Deeply nested schemas are harder for models to satisfy and harder for you to debug, because a single misplaced object can fail the whole validation with an unhelpful error. Where you can, flatten. A schema that returns a list of typed records often outperforms one that returns a single deeply nested tree, and it is far easier to validate incrementally.
Decompose Instead of Mega-Schema
When a task genuinely needs rich structure, consider splitting it across multiple model calls rather than one heroic schema. Extract the top-level fields first, then make targeted follow-up calls for the complex sub-objects. You trade a little latency for a large gain in reliability and debuggability. The Trade-offs, Options, and How to Decide piece frames when this decomposition is worth the extra round trips.
Union Types and Conditional Fields
Discriminated Unions
A common hard case is a field that can be one of several shapes — an event that is either a purchase, a refund, or a cancellation, each with different fields. Model these as discriminated unions with an explicit type tag. The tag tells both the model and your validator which branch applies, and it makes the resulting data easy to switch on in code. Without the tag, you are guessing at the shape after the fact.
Conditional Requirements
Sometimes a field is required only when another field has a certain value. Some schema systems express this directly; where they do not, encode the rule in your business-rule validation layer rather than hoping the model infers it. Be explicit in the prompt as well, because the model handles conditional logic better when it is stated than when it is implied by the schema alone.
Streaming Structured Output
Validating Before the Object Is Complete
Streaming a structured response into a UI is powerful and treacherous. You cannot validate a partial JSON object against a strict schema, because it is, by definition, incomplete. The pattern is to use a tolerant incremental parser that yields partial objects as fields complete, render optimistically, and run full validation only once the stream closes. Treat everything shown mid-stream as provisional.
Guarding the User Experience
Decide what the UI does if the completed object fails final validation after you have already shown partial content. Often the right move is to show progress optimistically but commit nothing — no database write, no irreversible action — until the final validation passes. Streaming is for perceived responsiveness, not for committing partial truth. The Best Practices That Actually Work piece covers the commit-on-validate discipline in more depth.
Partial Recovery From Near-Miss Output
A response that fails validation is not always worthless. Often it is correct except for one field. Blindly retrying the whole call wastes tokens and may fail the same way.
- Diagnose the specific failure from the validator error rather than treating all failures alike.
- Repair narrowly when the fix is mechanical — a stray field, a type that needs coercion — rather than regenerating everything.
- Re-ask for one field when a single value is missing or implausible, passing the rest back as context, instead of redoing the full extraction.
- Escalate to a human only after cheaper recovery paths fail.
This graduated recovery is far more efficient than a blunt retry loop and noticeably cheaper at scale.
Evaluating Structured Output Like You Mean It
Build a Labeled Set
Advanced reliability is not vibes; it is measurement. Assemble a set of representative inputs with known-correct structured outputs, including the awkward edge cases that break naive setups. Run candidate prompts, schemas, and models against it and score conformance and semantic correctness separately.
Test the Distribution, Not the Happy Path
The inputs that break structured output cluster — unusual languages, very long documents, ambiguous content, adversarial phrasing. Deliberately oversample these in your evaluation set so your reliability number reflects the hard cases rather than the easy ones. A conformance rate measured only on clean inputs is a comforting lie. The Real-World Examples and Use Cases collection is a good source of awkward inputs worth adding.
Handling Large Arrays and Long Output
The Truncation Trap
A schema that returns a long array invites a quiet failure mode: the model runs into the output token limit mid-array, and you get a valid-looking response that is simply incomplete. Strict decoding does not save you here, because a truncated array can still parse as valid JSON up to the cutoff. The fix is to detect truncation explicitly — check whether the response hit the length limit — and treat it as a failure rather than trusting a partial list.
Pagination and Chunking
When the natural output is genuinely large, do not force it into one call. Process the input in chunks and accumulate structured results, or ask the model for a bounded number of items per call with a cursor. This keeps each response well within limits and within the model's reliable range. The reliability of a structured call degrades as the expected output grows, so bounding output length is one of the highest-leverage moves available. The Best Tools for Structured Output and JSON Mode piece notes which libraries help manage this chunking automatically.
Ordering and Determinism
Array output is not guaranteed to come back in a stable order across calls, which breaks any downstream logic that assumes positional meaning. If order matters, either make it explicit in the schema with an index field or sort deterministically after extraction. Never lean on incidental ordering; it is exactly the kind of assumption that holds in testing and fails in production.
Expert Nuances Worth Internalizing
A few hard-won points that experienced practitioners converge on:
- Strict decoding can mask prompt problems. The output conforms, so you assume the prompt is good, while the values are subtly wrong. Always check meaning, not just shape.
- Larger schemas raise token cost and can lower quality. Trim relentlessly; every field you remove is one fewer thing to go wrong.
- Model upgrades change behavior. A schema that was reliable on one model version can regress on the next, even an ostensibly better one. Re-evaluate on upgrade.
- Enums beat free text every time a field has a bounded value set, both for reliability and for downstream code.
Frequently Asked Questions
When should I split one schema into multiple model calls?
When a single schema gets deeply nested or large enough that conformance drops, decompose it. Extract top-level fields first, then make focused follow-up calls for complex sub-objects. You accept extra latency in exchange for a meaningful reliability and debuggability gain.
How do I validate streaming structured output?
You cannot strictly validate an incomplete object. Use a tolerant incremental parser to render partial fields optimistically, then run full schema and business-rule validation only when the stream closes. Commit nothing irreversible until that final validation passes.
What is the most efficient way to recover from a near-miss response?
Diagnose the specific validation failure and repair narrowly. Coerce a type, drop a stray field, or re-ask for the single problematic field with the rest passed back as context. Reserve full regeneration and human escalation for cases where targeted repair fails.
Why does my schema regress after a model upgrade?
Model behavior shifts between versions, including how reliably it follows a given schema. A newer, generally better model can be worse on your specific structure. Treat every model change as a trigger to re-run your evaluation set before promoting it.
How do I keep deeply nested schemas reliable?
Flatten where you can, prefer lists of typed records over deep trees, and decompose genuinely complex structures across calls. Depth is the enemy of both reliability and debuggability, so spend it only where the use case truly requires it.
Key Takeaways
- Flatten schemas and decompose complex extractions across calls; depth costs reliability and debuggability.
- Model variant shapes as discriminated unions with explicit type tags so both the model and your code can branch cleanly.
- Stream optimistically but validate on completion and commit nothing irreversible until final validation passes.
- Recover from near-misses with narrow, graduated repair rather than blunt full retries.
- Evaluate against a labeled set that oversamples hard inputs, and re-evaluate on every model upgrade.