BroadRouter
Get started

JSON mode vs json_schema: structured output that holds up

By the BroadNet Inference team · 4 min read · Published

JSON mode (response_format json_object) only promises valid JSON. Structured output (response_format json_schema) also promises your fields and types, where the provider enforces it. Either way, check finish_reason, parse, and validate against a schema such as a Pydantic model. If validation fails, send the error back and ask again, with a hard limit on attempts.

Your code needs data, and models write text. Structured output closes that gap, but “the model returns JSON” means three different things depending on how you ask. This post compares the options on OpenAI-compatible open models, shows how to request each, and gives a validate-and-repair loop you can drop into production.

What is the difference between JSON mode and json_schema?

Method Request What you’re promised
Prompt only “Reply in JSON with fields…” Nothing; usually JSON, sometimes with extra text
JSON mode response_format: {"type": "json_object"} Valid JSON, any shape
Structured output response_format: {"type": "json_schema", ...} JSON matching your schema, where enforced

OpenAI’s Structured Outputs guide puts it directly: “While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence.” It recommends structured output over JSON mode whenever possible.

On open-model providers, the guarantee depends on how the server implements it. Many use constrained decoding, which blocks tokens that would break the schema. Some accept the field but don’t enforce it. That’s why the code below validates regardless.

How do you request json_schema?

import os
from openai import OpenAI
from pydantic import BaseModel, Field

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

class Invoice(BaseModel):
    vendor: str
    currency: str = Field(pattern="^[A-Z]{3}$")
    total: float
    line_items: list[float]

schema = Invoice.model_json_schema()
schema["additionalProperties"] = False

response_format = {"type": "json_schema",
                   "json_schema": {"name": "invoice", "strict": True, "schema": schema}}

Pydantic generates the JSON Schema from your class, so the schema and the validator never drift apart. If you don’t use Pydantic, the JSON schema generator builds a strict-mode schema from a sample object. For strict mode, OpenAI requires additionalProperties: false and every property listed as required.

Which mode does your provider support?

Run this test once per model before you build on it:

  1. Send json_schema with a prompt that invites an extra field (“also add a short comment”).
  2. If the request errors, the provider doesn’t accept json_schema. Try json_object.
  3. If the reply contains the extra field, the schema isn’t enforced. Treat it as JSON mode.
  4. If json_object also errors, use prompt-only with validation.

On BroadNet Inference, the docs list structured output for GLM 5.3 and Qwen3.8 27B (compared side by side), but don’t name the request parameter. Run the test above on the model you ship.

How do you validate and repair bad output?

import json
from pydantic import ValidationError

def extract(text: str, model="broadnet/glm-5.3", attempts=3) -> Invoice:
    messages = [
        {"role": "system", "content": "Extract the invoice. Reply with JSON only."},
        {"role": "user", "content": text},
    ]
    for attempt in range(attempts):
        r = client.chat.completions.create(model=model, messages=messages,
                                           response_format=response_format,
                                           max_tokens=4000)
        choice = r.choices[0]
        if choice.finish_reason == "length":
            raise RuntimeError("Cut off: raise max_tokens")  # retrying won't help
        if getattr(choice.message, "refusal", None):
            raise RuntimeError(f"Refused: {choice.message.refusal}")
        raw = choice.message.content or ""
        try:
            inv = Invoice.model_validate_json(raw)
            if abs(sum(inv.line_items) - inv.total) > 0.01:
                raise ValueError("total must equal the sum of line_items")
            return inv
        except (ValidationError, ValueError) as e:
            messages += [
                {"role": "assistant", "content": raw},
                {"role": "user", "content": f"That JSON is invalid: {e}. Return corrected JSON only."},
            ]
    raise RuntimeError(f"No valid JSON after {attempts} attempts")

What makes this loop work:

  • Check finish_reason first. A reply that hit max_tokens is truncated JSON. Retrying with the same limit fails the same way. Both models above reason before answering, and reasoning counts toward the limit, so leave room.
  • Handle refusals separately. OpenAI’s guide notes a refusal “does not necessarily follow the schema you have supplied”.
  • Send the exact error back. A blind retry often returns the same mistake. “line_items: field required” gives the model something to fix.
  • Check business rules in code. A schema can’t say a total must equal its line items. The check above can.
  • Cap attempts. Every attempt resends the whole prompt as input tokens.

What schema features cause trouble?

Keep schemas boring:

  1. Flat beats nested. Two levels is plenty for most extraction.
  2. Enums over free strings for categories.
  3. No open-ended objects. Free-form dict fields defeat the point.
  4. Short field names with descriptions. Descriptions in the schema guide the model.
  5. Optional fields as nullable. In strict mode, list every field as required and allow null.

OpenAI’s guide notes some JSON Schema features are unsupported in strict mode. Open-model servers have their own limits. If a schema is rejected, simplify it before you switch modes.

How do you monitor structured output in production?

A loop that repairs silently can hide a problem that costs you tokens every day. Log three numbers per call site:

Metric What it tells you
First-attempt success rate How well the model and schema fit; a drop often follows a prompt or model change
Average attempts per success The token cost of repairs
Final failures Inputs your pipeline can’t handle; review them by hand

If first-attempt success falls below what you’re used to, simplify the schema or try another model before you add more retries. Keep a sample of failed raw replies, so you can test fixes against real cases.

When should you use a tool instead?

Use structured output for the final answer: an extraction, a classification, a plan. Use tool calling when the model needs your code to do something and continue. Many agents use both: tools during the loop, a schema for the last step. For an image-to-JSON example, see reading Arabic invoices with a vision model.

Where BroadNet Inference fits

BroadNet Inference lists structured output for GLM 5.3 and Qwen3.8 27B, behind the standard chat-completions format, so the code above runs with the OpenAI Python SDK unchanged. Qwen3.8 27B also takes images, which suits document extraction. BroadNet’s docs don’t state structured output for DeepSeek V4.1 Flash or Gemma 4 VL 26B, so use the prompt-only path with validation there. Compare per-token prices on the pricing page.

Questions

What is the difference between JSON mode and structured output?

JSON mode, response_format type json_object, only promises the reply parses as JSON. Structured output, type json_schema, also promises the fields and types you defined, where the provider enforces it. OpenAI recommends structured output over JSON mode when possible.

Do open models support json_schema response_format?

It depends on the provider. Some accept json_schema, some only json_object, and some ignore the field. Send a test request with a schema and a prompt that tempts the model to add an extra field, then check the reply.

Do I still need to validate structured output?

Yes. Replies can be cut off by max_tokens, can be refusals, and can break business rules a schema can't express, such as totals that must add up.

How many times should I retry invalid JSON?

Two or three attempts with the validation error included is usually enough. After that, log the input and route it to a fallback, a stronger model or a person. Unbounded loops waste tokens.

Should I use structured output or tool calling?

Use structured output when you want the final answer in a fixed shape. Use tool calling when the model needs your code to do something and then continue.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity