LLM Structured Outputs: Get Reliable JSON

Many business uses of language models need data, not prose: fields pulled from an email, a classification for a support ticket, line items from an invoice, a set of arguments for a function. Asking a model to "reply in JSON" works most of the time, which is exactly the problem. The occasional missing field, extra comment or invented category breaks the code that consumes it, usually at the worst moment. LLM structured outputs address this by constraining the model to a schema you define. They make the format reliable, though not the content. This guide explains how structured outputs work, how to design schemas for them, why validation is still needed and how to handle refusals and messy inputs.

What LLM structured outputs guarantee

OpenAI's documentation describes Structured Outputs as a feature that ensures the model will always generate responses that adhere to your supplied JSON Schema, so you do not need to worry about the model omitting a required key or producing an invalid enum value (OpenAI: Structured model outputs). It describes Structured Outputs as the evolution of JSON mode: both produce valid JSON, but only Structured Outputs ensures schema adherence. Other providers offer comparable features; Anthropic's documentation, for example, covers strict tool use.

The same OpenAI guide is clear about the limit: Structured Outputs can still contain mistakes. The schema guarantees shape, not truth. A field typed as a date will contain a well-formed date, but not necessarily the right one.

Design schemas for the task

A good schema makes the task easier for the model and the output easier for your code:

• Keep it as small as the task needs; every extra field is another chance for a wrong value
• Use enumerations for fields with a fixed set of values, such as ticket categories or priority levels
• Describe each field, including format and meaning, in the schema's descriptions
• Prefer flat structures where possible; they are easier to review, validate and map to database columns
• Add a field for confidence or a reason only if your code will actually use it

The OpenAI guide lists specific schema rules for its implementation. For example, all fields must be marked as required, though an optional value can be emulated with a type that allows null, and objects must set additionalProperties to false. Check the current rules for the provider and model you use, because supported schema features differ.

Keep schemas and code in step

When the schema lives in one place and the data classes your code uses live in another, they drift apart. The OpenAI guide strongly recommends using the native schema helpers in its SDKs where available, or adding checks in continuous integration that flag when either the schema or the data types change. The guide also suggests a CI step that generates the JSON Schema from type definitions or the reverse, which removes the chance of drift. In Python, a validation library can define the data model once and produce the JSON Schema from it.

Validate business rules after parsing

Because the content can still be wrong, validate every response against business rules before using it:

• Cross-field checks, such as line item totals adding up to the invoice total
• Reference checks, such as a customer ID that exists in the CRM
• Range checks, such as dates within a plausible period
• Consistency with the source, such as an extracted amount that appears in the original text

Responses that fail go to a review queue or trigger a retry with a clearer instruction. Our guide to AI invoice data extraction describes this kind of validation in a document pipeline.

Handle refusals and incompatible inputs

Sometimes the model should not fill the schema at all: the input is not the expected type of document, the request is out of scope or the model declines for safety reasons. The OpenAI guide lists explicit refusals as a benefit, meaning safety-based refusals are programmatically detectable rather than hidden inside a malformed answer. It also suggests instructing the model to return empty parameters, or a specific sentence, if it detects that the input is incompatible with the task.

Design for these cases from the start. Add a status field, such as ok, not_applicable or needs_review, and route each status to the right handling in your code.

Worked example: a hypothetical extraction schema

The table below outlines a hypothetical schema for classifying inbound sales emails. It illustrates design choices, not a tested schema.

FieldTypeAllowed values or formatWhystatusEnumok, not_a_sales_email, needs_reviewRoutes incompatible inputsintentEnumpricing, demo_request, support, otherFixed categories for routingcompany_nameString or nullExactly as written in the emailNull when not statedrequested_dateString or nullISO dateNull when no date givensummaryStringOne sentenceShown to the sales rep

After parsing, code checks that requested_date is in the future, looks up company_name in the CRM and sends needs_review cases to a person.

Version the schema

Treat the schema as a contract between the model call and the code that consumes it. Give it a version, change it through review and update the consuming code, the prompt and the tests in the same change. Store the schema version with each saved result, so records produced under an older version can be recognised and migrated if needed.

Use structured outputs with tools

Structured outputs and tool calling are closely related. When a model calls a function, its arguments follow the function's schema; strict schema adherence for tool arguments prevents malformed calls reaching your code. Our guide to LLM tool calling covers how to design and validate those calls. Use a structured response format when the model's answer is the data you need, and tool calls when the model is asking your application to do something.

Test with messy inputs

Clean test examples hide problems. Build a test set from real inputs, with personal data removed: forwarded email chains, scanned documents, mixed languages, missing information and inputs that do not fit the task at all. Check not only that outputs parse, but that values are correct and that statuses route each case properly. Rerun the tests when you change the schema, the prompt or the model.

Watch production quality

Track how often responses fail business validation, how often they land in review and which fields are most often corrected by people. These numbers show where the schema, instructions or examples need work. The OpenAI guide suggests adjusting instructions, adding examples or splitting tasks into simpler subtasks when you see mistakes.

Frequently Asked Questions

What are LLM structured outputs?

A feature that constrains a model's response to a JSON Schema you supply, so required fields are present and values match the allowed types and enumerations.

Do structured outputs make the content correct?

No. They guarantee the format, not the facts. OpenAI's documentation notes that structured outputs can still contain mistakes, so validate business rules before using the data.

How is this different from JSON mode?

JSON mode ensures valid JSON. Structured outputs also ensure the JSON matches your schema.

What should happen when the input does not fit the task?

Include a status field so the model can mark the input as not applicable or needing review, and route those cases separately in your code.

Turn model output into reliable data

We design schemas, validation and review flows that turn language model output into data your systems can trust. See our AI agent orchestration and RAG service, or talk to us about LLM structured outputs.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.