BroadRouter

How to fix invalid JSON from an LLM: causes, repair, retry

By the BroadNet Inference team · 7 min read · Published · Updated

Check finish_reason first: a reply cut off at max_tokens cannot be repaired safely, so raise the limit and resend. Otherwise strip code fences and surrounding prose, decode the first JSON value, then fix only unambiguous mistakes such as trailing commas. Validate against a schema, and if that fails, send the exact error back once or twice. Structured output prevents most of this where supported.

Your code calls json.loads() on a model reply and gets JSONDecodeError: Expecting value: line 1 column 1. Or it parses, and a field is a string where you wanted a number. This post sorts the causes by what you can safely do about each, gives a tested Python function that repairs only the unambiguous cases, and shows a retry loop that knows when a retry can’t help.

Choosing between JSON mode and json_schema is covered in JSON mode vs json_schema. This post is about what to do with the reply you actually got.

Why do LLMs return invalid JSON?

A model writes text one token at a time. Unless the server constrains decoding, nothing stops it adding a friendly sentence or running out of room. The common failures:

What you see Cause Safe to repair in code?
```json … ``` around the object Chat formatting habit Yes: strip the fence
“Here is the JSON:” before, “Let me know…” after Same habit Yes: read the first value only
<think>...</think> before the object Reasoning text in the content field on some servers Yes: remove the block
Ends mid-string, brackets left open Hit max_tokens No: resend with more room
Empty content Reasoning used the whole budget No: resend with more room
{"a": 1,} Trailing comma; JavaScript allows it, RFC 8259 does not Yes: one meaning
{'a': True, 'b': None} Python syntax Yes: ast.literal_eval reads it safely
NaN, Infinity Not JSON; Python accepts them by default No: reject
Valid JSON, wrong types or missing fields Schema not enforced No: re-ask with the error

The dividing line is ambiguity. If there is exactly one thing the model could have meant, fix it in code and log that you did. If you would have to guess, reject the reply.

Why not repair truncated JSON?

Because the repaired version looks right and isn’t. We ran json_repair 0.64.9, a popular MIT-licensed library, on a reply cut off mid-word:

'{"name": "Omar", "city": "Jed'  ->  {'name': 'Omar', 'city': 'Jed'}

It returned valid JSON with the city “Jed”. Its README is open about this: it “automatically completes missing values in JSON fields with reasonable defaults”. That is useful for reading logs. For data you store or act on, it turns a visible failure into a silent one.

Check finish_reason before you parse. "length" means the reply hit max_tokens. Resending the same request with the same limit fails the same way, so double the limit or ask for less per call. Reasoning models make this more common: BroadNet’s API reference says DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before they answer, and the reasoning counts toward max_tokens. With a small limit, the answer can come back empty. Leave 1,000 tokens or more, or set reasoning_effort to low; the reasoning budget post has the numbers.

A repair function that only fixes the safe cases

This uses only the standard library. It returns the parsed value and a list of the repairs it made, so you can count them.

import ast, json, re

class Truncated(Exception): pass
class NotJSON(Exception): pass

FENCE = re.compile(r"```(?:json|JSON)?\s*\n(.*?)```", re.S)
THINK = re.compile(r"<think>.*?</think>", re.S)

def _reject_constant(name):                 # NaN, Infinity: Python accepts them, JSON does not
    raise NotJSON(f"non-standard constant {name}")

_decoder = json.JSONDecoder(parse_constant=_reject_constant)

def _first_value(s):
    """Decode the first JSON object or array in s, ignoring prose before and after it."""
    for i, ch in enumerate(s):
        if ch in "{[":
            try:
                return _decoder.raw_decode(s, i)[0]
            except json.JSONDecodeError:
                continue                    # a stray brace in prose; try the next one
    raise NotJSON("no JSON object or array found")

def _scan(s):
    """Drop commas before } or ] outside strings; report brackets left open."""
    out, stack, in_str, esc = [], [], False, False
    for i, ch in enumerate(s):
        if in_str:
            if esc:
                esc = False
            elif ch == "\\":
                esc = True
            elif ch == '"':
                in_str = False
        elif ch == '"':
            in_str = True
        elif ch in "{[":
            stack.append(ch)
        elif ch in "}]" and stack:
            stack.pop()
        elif ch == ",":
            rest = s[i + 1:].lstrip()
            if rest[:1] in ("}", "]"):
                continue                    # trailing comma: skip it
        out.append(ch)
    return "".join(out), len(stack) + (1 if in_str else 0)

def _python_literal(s):
    """Output in Python syntax: single quotes, True/False/None, trailing commas."""
    start = min((i for i in (s.find("{"), s.find("[")) if i >= 0), default=-1)
    end = max(s.rfind("}"), s.rfind("]"))
    if start < 0 or end <= start:
        raise NotJSON("no braces")
    value = ast.literal_eval(s[start:end + 1])       # literals only; never runs code
    if not isinstance(value, (dict, list)):
        raise NotJSON("not an object or array")
    json.dumps(value, allow_nan=False)               # must round-trip as strict JSON
    return value

def parse_llm_json(content, finish_reason=None):
    """Return (value, repairs). Raises Truncated or NotJSON."""
    if finish_reason == "length":
        raise Truncated("hit max_tokens; raise the limit or shorten the output")
    if not content or not content.strip():
        raise NotJSON("empty content (reasoning may have used the whole budget)")
    repairs = []
    s = content.strip().lstrip("\ufeff")
    if THINK.search(s):
        s = THINK.sub("", s).strip(); repairs.append("removed <think> block")
    m = FENCE.search(s)
    if m:
        s = m.group(1).strip(); repairs.append("removed code fence")
    try:
        value = _decoder.decode(s)
        return value, repairs
    except json.JSONDecodeError:
        pass
    try:
        value = _first_value(s); repairs.append("ignored text around JSON")
        return value, repairs
    except NotJSON:
        pass
    fixed, open_count = _scan(s)
    if open_count:
        raise NotJSON(f"incomplete JSON, {open_count} bracket(s) or string left open: treat as truncated")
    if fixed != s:
        try:
            value = _first_value(fixed); repairs.append("removed trailing commas")
            return value, repairs
        except NotJSON:
            pass
    try:
        value = _python_literal(s); repairs.append("parsed Python-style literal")
        return value, repairs
    except (ValueError, SyntaxError, TypeError, MemoryError, RecursionError, NotJSON) as e:
        raise NotJSON(f"unrepairable: {e}") from None

How each step works:

  1. Truncation and empty replies raise before any parsing, so they never become data.
  2. Fences and reasoning blocks are stripped with two regular expressions.
  3. raw_decode is the key tool. Python’s docs say it decodes a document “from a string that may have extraneous data at the end”, and returns where it stopped. Starting at the first { or [, it skips prose before the object and ignores prose after it. If a brace in prose fails to decode, it tries the next one.
  4. The bracket scan walks the text tracking strings and escapes. Brackets left open mean the reply was cut off even if finish_reason didn’t say so; that raises. Commas directly before } or ] outside strings are dropped.
  5. ast.literal_eval reads Python-style output with single quotes, True and None. It evaluates literals only and never runs code. The result must then serialise as strict JSON.
  6. NaN and Infinity raise, through parse_constant. Python’s json accepts them by default; RFC 8259 says they “are not permitted”, and JavaScript’s JSON.parse rejects them.

We ran it on 17 test strings: clean JSON, fenced JSON, Arabic keys and values in a fence, prose before and after, a {name} placeholder in prose before the object, a <think> block, trailing commas in objects and arrays, commas and brackets inside strings, escaped quotes, a Windows path ending in a backslash, Python literals, NaN, an empty reply, a reply cut off with and without finish_reason: "length", and two objects in a row. All behaved as the table above says. One test caught a real bug: an early version flipped its escape flag before checking for the closing quote, so "he said \"hi,]\"" lost its comma. Test your own repair code on escaped quotes.

Two limits to know. Given two objects in a row, it returns the first; if you want several items, ask for one JSON array. And single-quoted Python output that contains a double quote inside a string can be misread by the bracket scan and rejected. Rejecting is the safe failure.

How should you retry after repair fails?

Repair handles format. Validation handles content: right fields, right types, sane values. This loop validates with the jsonschema package and picks the retry by cause:

import json, os
from jsonschema import Draft202012Validator
from openai import OpenAI
from llm_json import parse_llm_json, Truncated, NotJSON

client = OpenAI(base_url=os.environ.get("BASE_URL", "https://inference.broadnet.ai/v1"),
                api_key=os.environ["BROADNET_API_KEY"])
SCHEMA = {"type": "object", "additionalProperties": False,
          "required": ["name", "city", "amount"],
          "properties": {"name": {"type": "string", "minLength": 1},
                         "city": {"type": ["string", "null"]},
                         "amount": {"type": "number", "minimum": 0}}}
validator = Draft202012Validator(SCHEMA)

def extract(text, model="broadnet/qwen3.8-27b", max_tokens=1500, attempts=3, log=print):
    messages = [{"role": "system", "content": "Extract name, city and amount. Reply with one JSON object only, no code fence. "
                                              "Schema: " + json.dumps(SCHEMA)},
                {"role": "user", "content": text}]
    for attempt in range(1, attempts + 1):
        r = client.chat.completions.create(model=model, messages=messages, max_tokens=max_tokens,
                                           reasoning_effort="low", temperature=0)
        c = r.choices[0]
        try:
            value, repairs = parse_llm_json(c.message.content, c.finish_reason)
            errors = [f"{'/'.join(map(str, e.path)) or '(root)'}: {e.message}" for e in validator.iter_errors(value)]
            if not errors:
                log({"attempt": attempt, "repairs": repairs, "ok": True})
                return value
            problem = "The JSON does not match the schema: " + "; ".join(errors)
        except Truncated:
            log({"attempt": attempt, "ok": False, "why": "truncated"})
            max_tokens *= 2                 # same prompt, more room; a re-ask would be cut off too
            continue
        except NotJSON as e:
            problem = f"That reply was not valid JSON ({e})."
        log({"attempt": attempt, "ok": False, "why": problem})
        messages += [{"role": "assistant", "content": c.message.content or ""},
                     {"role": "user", "content": problem + " Return the corrected JSON object only."}]
    raise RuntimeError(f"no valid JSON after {attempts} attempts")

The rules in that loop:

  • Truncated: resend the same messages with double the limit. Adding a “please fix” message would make the prompt longer and the output no shorter.
  • Not JSON, or wrong shape: send the exact error back. “amount: ‘1200’ is not of type ‘number’” gives the model something specific to fix. A bare retry often repeats the mistake.
  • Cap attempts at three. Every retry resends the whole conversation as input tokens. After three, log the input and route it elsewhere.
  • Log the repairs list. A rising count of “removed code fence” after a prompt change tells you the prompt got worse, even though nothing failed.

We ran this loop against a local mock server that played back scripted replies. A fenced reply with a trailing comma and "amount": "1200" as a string was repaired, failed validation, and was fixed on the second call with the error sent back. A reply cut off with finish_reason: "length" was resent with max_tokens raised from 1,500 to 3,000 and succeeded. The mock recorded the request each time, confirming the message count and limit the loop sent. A real model can fail differently, so run it on 100 of your own inputs before trusting the numbers.

Does structured output make this unnecessary?

It removes most of it. A server that constrains decoding to your schema cannot emit a fence, a trailing comma or a missing field. vLLM, a common open-model server, does this with the xgrammar or guidance backends. Two failures remain even then: truncation at max_tokens, and refusals. OpenAI notes that JSON mode output is valid “except for in some edge cases” and that your application “must detect and handle” output that is not a complete object. So keep the finish_reason check and the validator, even with json_schema.

What each approach prevents:

Approach Fences and prose Syntax errors Wrong shape Truncation
Prompt only (“reply with JSON”) Reduced Reduced Reduced No
JSON mode (json_object) Yes Yes No No
Structured output (json_schema, enforced) Yes Yes Yes No

With JSON mode, OpenAI also says you “must always instruct the model to produce JSON” in a message; otherwise it may emit whitespace until it hits the token limit. Other OpenAI-compatible servers vary, so test the mode on the model you ship. The test is in the structured output post.

To check a reply by hand, paste it into the LLM JSON output validator, or build a strict schema from a sample object with the JSON schema generator.

Where BroadNet Inference fits

BroadRouter is BroadNet’s site. BroadNet’s API reference lists structured output for GLM 5.3 and Qwen3.8 27B only, and does not name the request parameter, so test json_schema on the model before relying on it. On DeepSeek V4.1 Flash and Gemma 4 VL 26B, use prompt-only JSON with the repair function and validator above. Three of the four models reason before answering, so budget max_tokens for it. For bulk extraction, the batch runner post shows how to log finish_reason per item. Prompts and replies are not retained after each request.

Questions

Why does my LLM return JSON inside a markdown code block?

Chat models usually format code for people, and a code fence is how chat interfaces show it. Ask for "JSON only, no code fence", but strip fences in code anyway: the instruction lowers the rate, it does not remove it.

How do I fix JSONDecodeError on LLM output?

Look at finish_reason first. If it is "length", the reply was cut off: raise max_tokens and resend. If not, strip code fences and decode the first JSON value with raw_decode, which ignores text after it. Fix trailing commas, then validate. Re-ask with the error if it still fails.

Should I use a JSON repair library?

For logs and debugging, yes. For data you act on, be careful. Libraries such as json_repair complete missing values with defaults, so a truncated city "Jed" comes back as a valid "Jed". Reject truncated output and resend instead.

Why is my LLM JSON output cut off?

The reply hit max_tokens. On reasoning models such as DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B, reasoning counts toward that limit, so a long think can leave too little room. Raise the limit, set reasoning_effort low, or ask for less output per call.

Does JSON mode guarantee valid JSON?

Nearly. OpenAI says JSON mode output is valid JSON "except for in some edge cases", such as reaching the token limit, and it does not enforce your schema. Structured output with json_schema adds schema adherence where the provider enforces it. Validate either way.

Which BroadNet models support structured output?

BroadNet's API reference lists structured output for GLM 5.3 and Qwen3.8 27B, checked on 15 November 2026. It does not list it for DeepSeek V4.1 Flash or Gemma 4 VL 26B, so use prompt-only JSON with repair and validation on those.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity