Some LLM jobs aren’t chats. You have 50,000 support tickets to label, 200,000 product descriptions to translate, or a million reviews to score. Providers like OpenAI and Anthropic sell a Batch API for this. Many OpenAI-compatible APIs, BroadNet Inference among them, don’t. This post shows how to run the job yourself: work out how long it will take, write a runner that respects the limits, and make it safe to stop and restart.
BroadRouter is BroadNet’s own site, so the worked numbers use BroadNet’s published limits. The method works on any chat-completions API; swap in your provider’s numbers.
Does BroadNet Inference have a Batch API?
No. The API reference, checked on 15 November 2026, documents POST /v1/chat/completions, POST /v1/audio/transcriptions and GET /v1/models (a model list), plus account endpoints under /portal/. There is no batch or file-upload endpoint and no batch discount. A bulk job is a loop of normal requests.
Here is what a hosted Batch API gives you, so you know what you are rebuilding:
| OpenAI Batch API | Anthropic Message Batches | Your own runner | |
|---|---|---|---|
| Price | 50% off synchronous | 50% off | Normal price |
| Turnaround | Within 24 hours | Most within 1 hour; expires after 24 | Set by your rate limits |
| Size per batch | 50,000 requests or 200 MB | 100,000 requests or 256 MB | Any size |
| Rate limits | Separate from per-model limits | Apply to queued requests | Your key’s limits |
| Result order | May differ; match on custom_id |
Match on custom_id |
You key results by ID |
Two lessons carry over. Both providers make you tag each request with an ID, because results don’t come back in order. Neither promises an answer in seconds. Your runner should work the same way: IDs on everything, and a plan measured in hours.
If a 50% discount matters more than turnaround, and the models you need are there, a hosted Batch API may be the cheaper tool. Price both with the LLM API cost calculator before you build.
How long will the job take?
Work this out before you write code. Two limits cap throughput, and the slower one wins:
- In-flight limit ÷ latency. With 4 requests in flight and 2 seconds per request, you can finish 2 a second.
- Requests per minute. At 30 a minute, you can start 0.5 a second, however fast the model is.
A BroadNet pay-as-you-go key allows 4 requests in flight and 30 a minute. At the values in the API reference when we checked (30 a minute and 4 in flight, 2026-11-15), the rate limit binds unless each request takes more than 8 seconds, and the table below uses 30 a minute:
| Items | One item per request | 20 items per request |
|---|---|---|
| 10,000 | 5.6 hours | 17 minutes |
| 100,000 | 2.3 days | 2.8 hours |
| 1,000,000 | 23 days | 28 hours |
At 30 requests a minute, one key does 1,800 requests an hour, or 43,200 a day. Retries count too: every 429 you trigger spends a slot.
The second column is the biggest lever. Short items, such as reviews or ticket subjects, can go 10 to 50 per request as a numbered list, with a JSON array back. Check that the array has one entry per input and that each carries its number. The Arabic text classification post shows the pattern. Long items won’t pack: each request still has to fit the model’s context window, and quality can drop as the list grows. Test packing on 200 items before you trust it.
Don’t spread one job across several keys to beat a per-key limit without asking. BroadNet’s Acceptable Use Policy forbids circumventing rate, concurrency or budget limits. If you need more, ask BroadNet, or use plan sessions, below. The batch job planner does this arithmetic for your item count, latency and limits.
What does a resumable runner look like?
This runner reads items.jsonl (one {"id", "text"} object per line) and appends one result line per item to results.jsonl. It needs only the OpenAI Python SDK.
import asyncio, json, os, random, time
from openai import AsyncOpenAI, APIStatusError, APIConnectionError, APITimeoutError
client = AsyncOpenAI(base_url=os.environ.get("BASE_URL", "https://inference.broadnet.ai/v1"),
api_key=os.environ["BROADNET_API_KEY"],
max_retries=0, # we retry ourselves, with a cap we can see
timeout=120)
MODEL = os.environ.get("MODEL", "broadnet/deepseek-v4.1-flash")
IN_FLIGHT = int(os.environ.get("IN_FLIGHT", "4")) # PAYG key: 4 in flight. Plan: 1 per session.
RPM = int(os.environ.get("RPM", "30")) # PAYG key: 30 requests a minute
MAX_ATTEMPTS = 6
class Pacer:
"""Spaces request starts at least 60/RPM seconds apart, across all workers."""
def __init__(self, rpm):
self.gap, self.next, self.lock = 60 / rpm, 0.0, asyncio.Lock()
async def wait(self):
async with self.lock:
now = time.monotonic()
delay = max(0.0, self.next - now)
self.next = max(now, self.next) + self.gap
await asyncio.sleep(delay)
def retryable(e):
if isinstance(e, (APIConnectionError, APITimeoutError)):
return True
return isinstance(e, APIStatusError) and e.status_code in (429, 500, 502, 503, 504)
async def call(item, pacer):
for attempt in range(MAX_ATTEMPTS):
await pacer.wait() # every attempt counts against the rate limit
try:
r = await client.chat.completions.create(
model=MODEL, max_tokens=1500, reasoning_effort="low",
messages=[{"role": "system", "content": "Classify the review as positive, negative or mixed. Reply with one word."},
{"role": "user", "content": item["text"]}])
c = r.choices[0]
return {"id": item["id"], "ok": True, "finish_reason": c.finish_reason,
"output": c.message.content, "usage": r.usage.model_dump() if r.usage else None}
except Exception as e:
if not retryable(e) or attempt == MAX_ATTEMPTS - 1:
return {"id": item["id"], "ok": False, "error": repr(e)[:300]}
await asyncio.sleep(random.uniform(0, min(60, 2 ** attempt))) # full jitter, capped
async def run(in_path, out_path):
done = set()
if os.path.exists(out_path): # resume: skip ids that already succeeded
with open(out_path) as f:
done = {json.loads(l)["id"] for l in f if l.strip() and json.loads(l).get("ok")}
todo = [it for it in map(json.loads, open(in_path)) if it["id"] not in done]
print(f"{len(done)} done, {len(todo)} to go")
pacer, queue = Pacer(RPM), asyncio.Queue()
for it in todo:
queue.put_nowait(it)
with open(out_path, "a") as out:
async def worker():
while True:
try:
it = queue.get_nowait()
except asyncio.QueueEmpty:
return
res = await call(it, pacer)
out.write(json.dumps(res, ensure_ascii=False) + "\n")
out.flush() # one line per finished item = the checkpoint
await asyncio.gather(*(worker() for _ in range(IN_FLIGHT)))
if __name__ == "__main__":
import sys
asyncio.run(run(sys.argv[1], sys.argv[2]))
Run it with python batch_run.py items.jsonl results.jsonl. We tested this exact file against a local mock server that returned random 429s (8%) and 500s (5%) and refused anything over 4 in flight or 30 starts a minute. 25 items finished in 55 seconds with 28 requests (3 retries), and never more than 30 starts in any 60-second window. With the mock’s limit raised to 300 a minute to save time, a 120-item run killed after 10 seconds resumed from 39 saved results and finished with 120 unique IDs and no duplicates.
Why a worker pool and not asyncio.gather?
The 429 guide uses a semaphore around asyncio.gather. That is fine for a few hundred calls. For a large job, a fixed pool of workers pulling from a queue is better:
- Memory.
gathercreates one task per item up front. A pool holds only the items in flight. - Concurrency is the pool size. Four workers can never have five requests open.
- Results stream to disk as they finish, in whatever order, keyed by ID.
The Pacer handles the second limit. A semaphore alone stops you exceeding 4 in flight, but if requests take one second, four workers would start 240 a minute and hit 429 on most of them. The pacer spaces starts 2 seconds apart, across all workers, retries included.
How do retries stay safe?
Retry only what can succeed on a second try:
| Response | Action |
|---|---|
| 429 | Back off and retry; BroadNet’s docs say “use response details and bounded backoff” |
| 500, 502, 503, 504, timeouts, dropped connections | Retry with bounded exponential backoff |
| 400, 422 | Fix the request; retrying sends the same error |
| 401, 402, 403 | Stop the job; key, balance or policy needs attention |
| 413 | Shorten the item |
The runner turns off the SDK’s own retries (max_retries=0) so there is exactly one retry policy, with a visible cap. Backoff is exponential with full jitter, capped at 60 seconds, so workers that failed together don’t retry together. After six attempts the item is written as "ok": false and the job moves on. One bad item never stops a million.
A model call has no side effects, so delivery here is at least once: if the process dies mid-request, that item is sent again on restart, and you pay for it twice. That is the only waste. If the job also writes to another system, such as tickets or emails, make that write idempotent by item ID and do it in a separate pass over results.jsonl. The long-running agents post covers idempotent side effects in depth.
How does checkpointing work?
The output file is the checkpoint. Each finished item is one flushed line, so after a crash, a deploy or Ctrl-C, the next run reads the file, collects IDs with "ok": true and queues only the rest. Items that failed earlier are retried on the next run, and their new line is appended. When you read results, take the last line per ID.
Three habits make this hold up over days:
- Stable IDs. Use a database key or a hash of the input, never a line number that shifts when the input file changes.
- Check
finish_reason. A reply cut off atmax_tokensis"ok": truebut useless. DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before they answer, and reasoning counts toward max_tokens. The runner setsreasoning_effort="low", which BroadNet documents, and leaves 1,500 tokens. - Log usage per line. Summing
usagefromresults.jsonlgives the real token bill to compare with your estimate.
What does it cost: pay-as-you-go or a plan session?
On pay-as-you-go, cost is tokens. Take 100,000 reviews, a 150-token system prompt, a 150-token review and 100 output tokens with low reasoning. That is 30 million input and 10 million output tokens. On DeepSeek V4.1 Flash it costs 30 × $0.175 + 10 × $0.52. Keep the system prompt first and byte-identical, and its 15 million tokens can bill at the cached input rate of $0.0055 per 1M where the cache hits. Your usage lines show how often it did.
On a plan, tokens are free and time is the cost. A DeepSeek V4.1 Flash plan session “runs one request at a time”, so set IN_FLIGHT=1 per session. Throughput per session is 86,400 ÷ seconds per item, per day. BroadNet quotes typical speed as about 100 tokens per second, but your latency depends on prompt length and reasoning, so measure it on 50 items. If you measure 3 seconds an item, one session does 28,800 a day:
| Job | Sessions | Time at 3 s/item | Plan |
|---|---|---|---|
| 100,000 items | 1 | About 3.5 days | One Orbit week, $15 per session |
| 100,000 items | 4 | About 21 hours | Four Orbit weeks, one per session; or about 21 Meteor hours per session at $1 each |
| 1,000,000 items | 6 | About 6 days | Six Orbit weeks |
Plans are delivered “best effort under fair use”, and BroadNet doesn’t publish a per-minute request limit for plan keys; your account shows the limits that apply. Run RPM from those, not from guesswork. The session planner guide explains how to split sessions across keys.
Where BroadNet Inference fits
BroadNet Inference has no Batch API, so it suits bulk jobs when you would rather control the runner yourself and pay normal per-token prices, or fix the cost with DeepSeek V4.1 Flash plan sessions. For small jobs on a budget, Gemma 4 VL 26B at $0.10 input and $0.22 output per 1M runs on the $5 starting credit. Prompts and outputs are not retained after each request. If your job can wait a day and the models you need are on a provider with a 50% batch discount, that may cost less; compare before you commit. Plans and limits are on inference.broadnet.ai/plans.
