Tool calling is what turns a model into an agent. With open models on an OpenAI-compatible API, the request format is the same one you’d use with OpenAI. What differs is how often things go wrong and what the server does to prevent it. This post covers the format, how to write schemas open models handle well, parallel calls, the five failures you’ll actually see, and a Python loop that survives them.
How does tool calling work on an OpenAI-compatible API?
- You send
messagesplus atoolslist. Each tool has a name, a description and a JSON Schema for its arguments. - The model replies with
finish_reason: "tool_calls"and atool_callslist. Each entry has anid, a function name andargumentsas a JSON string. - Your code runs each call and appends one
role: "tool"message per call, with the matchingtool_call_id. - You call the API again. The model answers or asks for more tools.
The model never runs anything. Your code decides what to execute.
How should you write tool schemas?
Open models follow simple schemas more reliably. These rules come from OpenAI’s guide and from what tends to break:
- Name tools by action, such as
read_fileorsearch_code, and say in the description when to use each. - Keep the list short. OpenAI’s guide calls fewer than 20 functions at the start of a turn a soft suggestion. Every schema is also sent on every call; see the agent token budget.
- Prefer flat arguments. Strings, numbers, booleans and enums. Avoid deep nesting and free-form objects.
- Use enums for fixed choices, so the model can’t invent a value. The JSON schema generator turns a sample arguments object into a flat schema you can paste into
parameters. - Mark required fields. OpenAI’s strict mode requires
additionalProperties: falseand every field inrequired, withnullas a type for optional ones. Whether a provider enforcesstrictvaries, so validate anyway.
What about parallel tool calls?
A model may return several entries in tool_calls in one turn, for example three file reads. Run independent calls concurrently and return one result per tool_call_id, in any order. If your tools have side effects that depend on each other, set parallel_tool_calls: false; OpenAI’s guide says that limits the model to zero or one call per turn. Test that your provider honors it.
Why do tool calls fail?
| Failure | What you see | Fix |
|---|---|---|
| Malformed arguments | json.loads raises |
Return the parse error as a tool result; retry |
| Schema mismatch | Missing field, wrong type, invented enum | Validate; return the validation error |
| Unknown tool | A name you never defined | Return “unknown tool, available: …” |
| No call when one is needed | Text answer instead | tool_choice: "required" or a named function |
| Cut-off call | finish_reason: "length" |
Raise max_tokens; reasoning shares it |
The first two are expected, not rare. vLLM’s tool-calling docs say that in auto mode, unless a tool sets strict: true (or the server enforces a strictness floor) and the model’s parser supports constraints, “vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function’s parameter schema.” A named function uses vLLM’s structured-outputs backend. Your provider may serve models differently, so treat validation as mandatory.
The last row catches people using reasoning models. GLM 5.3 and Qwen3.8 27B think before they answer, and thinking counts toward max_tokens. Why replies come back empty or cut off explains how much room to leave.
A tool loop that validates and retries
import json, os
from openai import OpenAI
from jsonschema import validate, ValidationError
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"], max_retries=4)
MODEL = "broadnet/glm-5.3"
TOOLS = [{"type": "function", "function": {
"name": "get_order_status",
"description": "Look up one order by its ID. Use when the user asks where an order is.",
"parameters": {"type": "object",
"properties": {"order_id": {"type": "string"}},
"required": ["order_id"], "additionalProperties": False}}}]
SCHEMAS = {t["function"]["name"]: t["function"]["parameters"] for t in TOOLS}
IMPL = {"get_order_status": lambda order_id: f"Order {order_id}: shipped"}
def run_call(call):
name = call.function.name
if name not in IMPL:
return f"Error: unknown tool {name}. Available: {', '.join(IMPL)}"
try:
args = json.loads(call.function.arguments or "{}")
validate(args, SCHEMAS[name])
except (json.JSONDecodeError, ValidationError) as e:
return f"Error: invalid arguments ({e}). Send the call again with valid JSON."
return str(IMPL[name](**args))
def agent(messages, max_steps=8):
for _ in range(max_steps):
r = client.chat.completions.create(model=MODEL, messages=messages,
tools=TOOLS, max_tokens=4000)
msg = r.choices[0].message
if r.choices[0].finish_reason == "length":
raise RuntimeError("Output limit reached; raise max_tokens")
messages.append(msg.model_dump(exclude_none=True))
if not msg.tool_calls:
return msg.content
for call in msg.tool_calls:
messages.append({"role": "tool", "tool_call_id": call.id,
"content": run_call(call)})
raise RuntimeError("Step limit reached")
Three design choices matter here:
- Errors go back to the model as tool results. The model sees what was wrong and usually fixes it on the next turn. Raising an exception throws that chance away.
- The loop is bounded.
max_stepsstops a model that keeps calling tools without progress. - Transport retries are separate.
max_retrieson the client handles 429 and 5xx. Bad arguments are handled in the loop. Don’t mix the two; see errors and retries.
For tools with side effects, such as sending an email, add an idempotency key so a retried step doesn’t act twice. Long-running agents covers this.
Which BroadNet models support tool calling?
| Model | Tools listed | Context | Notes |
|---|---|---|---|
| GLM 5.3 | Yes | 1,048,576 | Also structured output |
| Qwen3.8 27B | Yes | 262,144 | Also image input |
| DeepSeek V4.1 Flash | Not listed | 1,048,576 | Image input; test tools before relying on them |
| Gemma 4 VL 26B | Not listed | 32,768 | Not for tool loops |
To choose between the two tool-capable models, see GLM 5.3 vs Qwen3.8 27B. When the final answer must match a shape but no function needs to run, structured output is simpler than a tool.
Where BroadNet Inference fits
BroadNet Inference serves GLM 5.3 and Qwen3.8 27B with tool calling listed, through the standard chat-completions format, so the loop above runs unchanged with the OpenAI Python SDK. Prices per token are on the pricing page. It doesn’t host server-side tools such as web search or code execution, and its docs don’t list tool calling for DeepSeek V4.1 Flash, the model on the unlimited plans. For a tool-heavy agent, use GLM 5.3 or Qwen3.8 27B per token. If you want the flat plan anyway, check on a $1 Meteor hour that DeepSeek V4.1 Flash calls your tools reliably before buying a month.
