BroadRouter
Get started

LLM batch processing without a Batch API: a resumable runner

By the BroadNet Inference team · 8 min read · Published · Updated

Without a Batch API, run the job yourself: read items from a JSONL file, cap requests in flight at the key's limit, pace request starts under the per-minute limit, retry 429 and 5xx with capped backoff, and append each result with its ID. On restart, skip finished IDs. The rate limit, not the model, usually sets the finish time.

Some LLM jobs aren’t chats. You have 50,000 support tickets to label, 200,000 product descriptions to translate, or a million reviews to score. Providers like OpenAI and Anthropic sell a Batch API for this. Many OpenAI-compatible APIs, BroadNet Inference among them, don’t. This post shows how to run the job yourself: work out how long it will take, write a runner that respects the limits, and make it safe to stop and restart.

BroadRouter is BroadNet’s own site, so the worked numbers use BroadNet’s published limits. The method works on any chat-completions API; swap in your provider’s numbers.

Does BroadNet Inference have a Batch API?

No. The API reference, checked on 15 November 2026, documents POST /v1/chat/completions, POST /v1/audio/transcriptions and GET /v1/models (a model list), plus account endpoints under /portal/. There is no batch or file-upload endpoint and no batch discount. A bulk job is a loop of normal requests.

Here is what a hosted Batch API gives you, so you know what you are rebuilding:

OpenAI Batch API Anthropic Message Batches Your own runner
Price 50% off synchronous 50% off Normal price
Turnaround Within 24 hours Most within 1 hour; expires after 24 Set by your rate limits
Size per batch 50,000 requests or 200 MB 100,000 requests or 256 MB Any size
Rate limits Separate from per-model limits Apply to queued requests Your key’s limits
Result order May differ; match on custom_id Match on custom_id You key results by ID

Two lessons carry over. Both providers make you tag each request with an ID, because results don’t come back in order. Neither promises an answer in seconds. Your runner should work the same way: IDs on everything, and a plan measured in hours.

If a 50% discount matters more than turnaround, and the models you need are there, a hosted Batch API may be the cheaper tool. Price both with the LLM API cost calculator before you build.

How long will the job take?

Work this out before you write code. Two limits cap throughput, and the slower one wins:

  • In-flight limit ÷ latency. With 4 requests in flight and 2 seconds per request, you can finish 2 a second.
  • Requests per minute. At 30 a minute, you can start 0.5 a second, however fast the model is.

A BroadNet pay-as-you-go key allows 4 requests in flight and 30 a minute. At the values in the API reference when we checked (30 a minute and 4 in flight, 2026-11-15), the rate limit binds unless each request takes more than 8 seconds, and the table below uses 30 a minute:

Items One item per request 20 items per request
10,000 5.6 hours 17 minutes
100,000 2.3 days 2.8 hours
1,000,000 23 days 28 hours

At 30 requests a minute, one key does 1,800 requests an hour, or 43,200 a day. Retries count too: every 429 you trigger spends a slot.

The second column is the biggest lever. Short items, such as reviews or ticket subjects, can go 10 to 50 per request as a numbered list, with a JSON array back. Check that the array has one entry per input and that each carries its number. The Arabic text classification post shows the pattern. Long items won’t pack: each request still has to fit the model’s context window, and quality can drop as the list grows. Test packing on 200 items before you trust it.

Don’t spread one job across several keys to beat a per-key limit without asking. BroadNet’s Acceptable Use Policy forbids circumventing rate, concurrency or budget limits. If you need more, ask BroadNet, or use plan sessions, below. The batch job planner does this arithmetic for your item count, latency and limits.

What does a resumable runner look like?

This runner reads items.jsonl (one {"id", "text"} object per line) and appends one result line per item to results.jsonl. It needs only the OpenAI Python SDK.

import asyncio, json, os, random, time
from openai import AsyncOpenAI, APIStatusError, APIConnectionError, APITimeoutError

client = AsyncOpenAI(base_url=os.environ.get("BASE_URL", "https://inference.broadnet.ai/v1"),
                     api_key=os.environ["BROADNET_API_KEY"],
                     max_retries=0,        # we retry ourselves, with a cap we can see
                     timeout=120)
MODEL = os.environ.get("MODEL", "broadnet/deepseek-v4.1-flash")
IN_FLIGHT = int(os.environ.get("IN_FLIGHT", "4"))     # PAYG key: 4 in flight. Plan: 1 per session.
RPM = int(os.environ.get("RPM", "30"))                # PAYG key: 30 requests a minute
MAX_ATTEMPTS = 6

class Pacer:
    """Spaces request starts at least 60/RPM seconds apart, across all workers."""
    def __init__(self, rpm):
        self.gap, self.next, self.lock = 60 / rpm, 0.0, asyncio.Lock()
    async def wait(self):
        async with self.lock:
            now = time.monotonic()
            delay = max(0.0, self.next - now)
            self.next = max(now, self.next) + self.gap
        await asyncio.sleep(delay)

def retryable(e):
    if isinstance(e, (APIConnectionError, APITimeoutError)):
        return True
    return isinstance(e, APIStatusError) and e.status_code in (429, 500, 502, 503, 504)

async def call(item, pacer):
    for attempt in range(MAX_ATTEMPTS):
        await pacer.wait()                          # every attempt counts against the rate limit
        try:
            r = await client.chat.completions.create(
                model=MODEL, max_tokens=1500, reasoning_effort="low",
                messages=[{"role": "system", "content": "Classify the review as positive, negative or mixed. Reply with one word."},
                          {"role": "user", "content": item["text"]}])
            c = r.choices[0]
            return {"id": item["id"], "ok": True, "finish_reason": c.finish_reason,
                    "output": c.message.content, "usage": r.usage.model_dump() if r.usage else None}
        except Exception as e:
            if not retryable(e) or attempt == MAX_ATTEMPTS - 1:
                return {"id": item["id"], "ok": False, "error": repr(e)[:300]}
            await asyncio.sleep(random.uniform(0, min(60, 2 ** attempt)))   # full jitter, capped

async def run(in_path, out_path):
    done = set()
    if os.path.exists(out_path):                    # resume: skip ids that already succeeded
        with open(out_path) as f:
            done = {json.loads(l)["id"] for l in f if l.strip() and json.loads(l).get("ok")}
    todo = [it for it in map(json.loads, open(in_path)) if it["id"] not in done]
    print(f"{len(done)} done, {len(todo)} to go")
    pacer, queue = Pacer(RPM), asyncio.Queue()
    for it in todo:
        queue.put_nowait(it)
    with open(out_path, "a") as out:
        async def worker():
            while True:
                try:
                    it = queue.get_nowait()
                except asyncio.QueueEmpty:
                    return
                res = await call(it, pacer)
                out.write(json.dumps(res, ensure_ascii=False) + "\n")
                out.flush()                         # one line per finished item = the checkpoint
        await asyncio.gather(*(worker() for _ in range(IN_FLIGHT)))

if __name__ == "__main__":
    import sys
    asyncio.run(run(sys.argv[1], sys.argv[2]))

Run it with python batch_run.py items.jsonl results.jsonl. We tested this exact file against a local mock server that returned random 429s (8%) and 500s (5%) and refused anything over 4 in flight or 30 starts a minute. 25 items finished in 55 seconds with 28 requests (3 retries), and never more than 30 starts in any 60-second window. With the mock’s limit raised to 300 a minute to save time, a 120-item run killed after 10 seconds resumed from 39 saved results and finished with 120 unique IDs and no duplicates.

Why a worker pool and not asyncio.gather?

The 429 guide uses a semaphore around asyncio.gather. That is fine for a few hundred calls. For a large job, a fixed pool of workers pulling from a queue is better:

  • Memory. gather creates one task per item up front. A pool holds only the items in flight.
  • Concurrency is the pool size. Four workers can never have five requests open.
  • Results stream to disk as they finish, in whatever order, keyed by ID.

The Pacer handles the second limit. A semaphore alone stops you exceeding 4 in flight, but if requests take one second, four workers would start 240 a minute and hit 429 on most of them. The pacer spaces starts 2 seconds apart, across all workers, retries included.

How do retries stay safe?

Retry only what can succeed on a second try:

Response Action
429 Back off and retry; BroadNet’s docs say “use response details and bounded backoff”
500, 502, 503, 504, timeouts, dropped connections Retry with bounded exponential backoff
400, 422 Fix the request; retrying sends the same error
401, 402, 403 Stop the job; key, balance or policy needs attention
413 Shorten the item

The runner turns off the SDK’s own retries (max_retries=0) so there is exactly one retry policy, with a visible cap. Backoff is exponential with full jitter, capped at 60 seconds, so workers that failed together don’t retry together. After six attempts the item is written as "ok": false and the job moves on. One bad item never stops a million.

A model call has no side effects, so delivery here is at least once: if the process dies mid-request, that item is sent again on restart, and you pay for it twice. That is the only waste. If the job also writes to another system, such as tickets or emails, make that write idempotent by item ID and do it in a separate pass over results.jsonl. The long-running agents post covers idempotent side effects in depth.

How does checkpointing work?

The output file is the checkpoint. Each finished item is one flushed line, so after a crash, a deploy or Ctrl-C, the next run reads the file, collects IDs with "ok": true and queues only the rest. Items that failed earlier are retried on the next run, and their new line is appended. When you read results, take the last line per ID.

Three habits make this hold up over days:

  1. Stable IDs. Use a database key or a hash of the input, never a line number that shifts when the input file changes.
  2. Check finish_reason. A reply cut off at max_tokens is "ok": true but useless. DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before they answer, and reasoning counts toward max_tokens. The runner sets reasoning_effort="low", which BroadNet documents, and leaves 1,500 tokens.
  3. Log usage per line. Summing usage from results.jsonl gives the real token bill to compare with your estimate.

What does it cost: pay-as-you-go or a plan session?

On pay-as-you-go, cost is tokens. Take 100,000 reviews, a 150-token system prompt, a 150-token review and 100 output tokens with low reasoning. That is 30 million input and 10 million output tokens. On DeepSeek V4.1 Flash it costs 30 × $0.175 + 10 × $0.52. Keep the system prompt first and byte-identical, and its 15 million tokens can bill at the cached input rate of $0.0055 per 1M where the cache hits. Your usage lines show how often it did.

On a plan, tokens are free and time is the cost. A DeepSeek V4.1 Flash plan session “runs one request at a time”, so set IN_FLIGHT=1 per session. Throughput per session is 86,400 ÷ seconds per item, per day. BroadNet quotes typical speed as about 100 tokens per second, but your latency depends on prompt length and reasoning, so measure it on 50 items. If you measure 3 seconds an item, one session does 28,800 a day:

Job Sessions Time at 3 s/item Plan
100,000 items 1 About 3.5 days One Orbit week, $15 per session
100,000 items 4 About 21 hours Four Orbit weeks, one per session; or about 21 Meteor hours per session at $1 each
1,000,000 items 6 About 6 days Six Orbit weeks

Plans are delivered “best effort under fair use”, and BroadNet doesn’t publish a per-minute request limit for plan keys; your account shows the limits that apply. Run RPM from those, not from guesswork. The session planner guide explains how to split sessions across keys.

Where BroadNet Inference fits

BroadNet Inference has no Batch API, so it suits bulk jobs when you would rather control the runner yourself and pay normal per-token prices, or fix the cost with DeepSeek V4.1 Flash plan sessions. For small jobs on a budget, Gemma 4 VL 26B at $0.10 input and $0.22 output per 1M runs on the $5 starting credit. Prompts and outputs are not retained after each request. If your job can wait a day and the models you need are on a provider with a 50% batch discount, that may cost less; compare before you commit. Plans and limits are on inference.broadnet.ai/plans.

Questions

Does BroadNet Inference have a Batch API?

No. Its API reference, checked on 15 November 2026, documents POST /v1/chat/completions, POST /v1/audio/transcriptions and GET /v1/models, with no batch or file-upload endpoint and no batch discount. Run bulk jobs as ordinary requests within your key's limits.

How do I process 100,000 prompts with an LLM API?

Put the items in a JSONL file with an ID each. Run a fixed number of workers equal to the key's in-flight limit, pace request starts under the per-minute limit, retry 429 and 5xx with capped backoff, and append each result to an output file keyed by ID so the job can resume.

How long will a batch of a million requests take?

Divide the items by your real rate. At 30 requests a minute and one item per request, one key does 43,200 a day, so a million takes about 23 days. Packing 20 short items per request cuts that to about 28 hours, if the model handles the longer prompt well.

Should I use asyncio.gather or a worker pool for batch LLM calls?

For tens of thousands of items, use a fixed pool of workers pulling from a queue. gather with a semaphore works for small jobs, but it creates every task up front and makes resuming and progress tracking harder.

Can I use several API keys to go faster?

Check first. BroadNet's Acceptable Use Policy forbids circumventing rate or concurrency limits, so ask BroadNet before you split one job across keys to get past a per-key limit. Packing more items per request and plan sessions are the documented ways to add throughput.

Is a flat plan good for batch jobs?

It can be, for DeepSeek V4.1 Flash. A session runs one request at a time with no token meter, so cost is fixed and the question is time. Measure seconds per item, then buy enough sessions to finish inside the period.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity