You wire a language model into your app, ask it to "respond in JSON," and it does. It works the first ten times. Then, on a Tuesday, it wraps the JSON in a markdown code fence, or opens with "Sure, here is the data," or trails off with a comment, and your parser throws, and the pager goes off. "Just ask nicely" is not a strategy. It is a demo that has not failed yet.
The reason it fails is structural. A base language model predicts likely text, and there are a thousand plausible ways to wrap a JSON object in friendly words. A tiny fraction of your calls will land on one of those, and a tiny fraction of a large number is a lot of broken responses. You need something stronger than a polite request.
Say what shape you want, exactly
The first move is to stop describing the output in prose and start defining it as a schema. A JSON Schema, or a typed object in your language that compiles to one, spells out the fields, their types, which are required, and what values are allowed. This does two things. It tells the model precisely what to produce, and it gives you something to validate against afterward, which matters more than people expect.
A schema also kills a whole class of quiet bugs. Without one, the model decides on a Tuesday to call the field "phone_number" instead of "phone," or returns the price as the string "nineteen dollars" instead of the number 19. With a strict schema those drifts either cannot happen or get caught immediately instead of three steps downstream.
Use the mode built for this
Most serious providers now offer a real structured-output feature, and you should use it instead of hoping. There are two flavors worth knowing.
- Structured outputs. You pass a schema and the model is constrained so its output conforms. OpenAI's version, launched in August 2024 with a strict flag, reported 100 percent adherence on their internal schema evals, up from under 40 percent for an older model that was merely asked. That gap is the whole argument.
- Function or tool calling. You describe a function with typed arguments, and the model returns a call to it with the arguments filled in. Same underlying idea, framed as calling code, and the right fit when the JSON is meant to trigger an action.
When the constrained mode is available, the model is not writing free text and then hoping it parses. The valid tokens are restricted as it generates, so malformed JSON becomes close to impossible. That is a completely different reliability story from prompt-and-pray, and it is why the failure rate on the good implementations sits at a fraction of a percent rather than a few percent.
Validate anyway, and retry on purpose
Constrained output handles the shape. It does not handle refusals, truncation from hitting a token limit, or values that are the right type but wrong in context, like a date of February 30. So the last piece is a loop you build yourself.
Parse the response. Validate it against the same schema you sent. If it passes, you are done. If it fails, do not just retry blind: feed the error back to the model, say what broke, and ask it to fix that specific thing. Cap the retries at two or three so a stubborn input cannot spin forever, and log the failures, because a cluster of them usually means your schema is asking for something ambiguous rather than the model misbehaving.
One detail that saves real pain: watch for the refusal path. The good structured-output APIs return a separate refusal field when the model declines, precisely so you do not try to parse an apology as data. Check it before you reach for the parser.
The version of this that holds up in production is boring, and boring is the compliment. A schema the model is constrained to follow, a validation step you never skip, and a short retry loop that tells the model what went wrong. Do that and JSON stops being the flaky part of your pipeline. Skip it, and you will meet the markdown code fence at the worst possible hour.