Every network call needs a point where you stop waiting. For LLM calls that point is hard to choose, because a slow answer and a dead connection look the same for a while.
How it works
HTTP clients split waiting into phases. HTTPX, a common Python HTTP client, has four:
| Timeout | HTTPX meaning | Error |
|---|---|---|
| Connect | Longest time to establish a socket connection | ConnectTimeout |
| Read | Longest wait for a chunk of incoming data | ReadTimeout |
| Write | Longest wait to send a chunk of the request body | WriteTimeout |
| Pool | Longest wait for a connection from the pool | PoolTimeout |
Plain HTTPX times out after five seconds of network inactivity by default. The OpenAI Python SDK sets its own default: “By default requests time out after 10 minutes”, raises APITimeoutError, and retries a timed-out request twice. It also retries connection errors, 408, 409, 429 and 5xx responses by default.
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", timeout=120.0, max_retries=2)
Servers have timeouts too. MDN defines 504 Gateway Timeout as a gateway or proxy that “did not get a response in time from the upstream server”, and 408 Request Timeout as the server wanting “to shut down this unused connection”.
Why it matters
- Reasoning models are slow to start. Thinking happens before the first visible token. A short read timeout on a non-streamed call can cut off a healthy request.
- Streams detect stalls. With streaming, each chunk resets the read timer, so a stall shows up fast even on a long answer.
- Retries can double cost. A timeout does not mean the server stopped. Retrying a request that is still running may bill twice. Use exponential backoff with a cap, and an idempotency key where an API supports one.
On BroadNet Inference
BroadNet’s API reference says to read a stream until data: [DONE]. If the connection ends before it, “treat the response as incomplete, record the X-Request-ID”, and decide whether your application can safely make a new request. For 500, 502, 503 and 504 it says to retry with bounded exponential backoff; 429 means a rate, concurrency, token-rate or spending limit was reached. Keep the X-Request-ID for support. The errors and retries guide and the long-running agents post show working settings.
Common mistakes
- Leaving the default and wondering why a hung call blocks a worker for ten minutes.
- Retrying without a cap, so one bad minute becomes a flood of requests.