BroadRouter
Get started

Request timeout

Updated

A request timeout is the longest time an HTTP client waits for a server before it abandons the request. Clients usually split it into a connect timeout for opening the connection and a read timeout for waiting on data. LLM calls need generous read timeouts because long answers take time.

Also called: read timeout, connect timeout, client timeout, HTTP timeout

Every network call needs a point where you stop waiting. For LLM calls that point is hard to choose, because a slow answer and a dead connection look the same for a while.

How it works

HTTP clients split waiting into phases. HTTPX, a common Python HTTP client, has four:

Timeout HTTPX meaning Error
Connect Longest time to establish a socket connection ConnectTimeout
Read Longest wait for a chunk of incoming data ReadTimeout
Write Longest wait to send a chunk of the request body WriteTimeout
Pool Longest wait for a connection from the pool PoolTimeout

Plain HTTPX times out after five seconds of network inactivity by default. The OpenAI Python SDK sets its own default: “By default requests time out after 10 minutes”, raises APITimeoutError, and retries a timed-out request twice. It also retries connection errors, 408, 409, 429 and 5xx responses by default.

from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", timeout=120.0, max_retries=2)

Servers have timeouts too. MDN defines 504 Gateway Timeout as a gateway or proxy that “did not get a response in time from the upstream server”, and 408 Request Timeout as the server wanting “to shut down this unused connection”.

Why it matters

  • Reasoning models are slow to start. Thinking happens before the first visible token. A short read timeout on a non-streamed call can cut off a healthy request.
  • Streams detect stalls. With streaming, each chunk resets the read timer, so a stall shows up fast even on a long answer.
  • Retries can double cost. A timeout does not mean the server stopped. Retrying a request that is still running may bill twice. Use exponential backoff with a cap, and an idempotency key where an API supports one.

On BroadNet Inference

BroadNet’s API reference says to read a stream until data: [DONE]. If the connection ends before it, “treat the response as incomplete, record the X-Request-ID”, and decide whether your application can safely make a new request. For 500, 502, 503 and 504 it says to retry with bounded exponential backoff; 429 means a rate, concurrency, token-rate or spending limit was reached. Keep the X-Request-ID for support. The errors and retries guide and the long-running agents post show working settings.

Common mistakes

  • Leaving the default and wondering why a hung call blocks a worker for ten minutes.
  • Retrying without a cap, so one bad minute becomes a flood of requests.

Questions

What is the default timeout in the OpenAI Python SDK?

Its README says "By default requests time out after 10 minutes", and timed-out requests are retried twice by default with a short exponential backoff.

Should I use one timeout for every LLM call?

No. A short classification call and a long reasoning answer need different limits. With streaming, a read timeout on the gap between chunks catches stalls better than one long total limit.

What should I do when a BroadNet stream stops early?

BroadNet says to treat a stream that ends before data: [DONE] as incomplete, record the X-Request-ID, and decide whether your application can safely make a new request.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity