---
title: "LLM batch processing without a Batch API - runner and math"
description: "Run 10,000 to 1,000,000 prompts through a chat-completions API with no Batch API: concurrency caps, rate pacing, retries, resumable checkpoints and the time math."
url: https://broadrouter.com/blog/llm-batch-processing-without-batch-api/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# LLM batch processing without a Batch API: a resumable runner

By the BroadNet Inference team · 8 min read · Published 2026-11-15 · Updated 2026-11-16

Without a Batch API, run the job yourself: read items from a JSONL file, cap requests in flight at the key's limit, pace request starts under the per-minute limit, retry 429 and 5xx with capped backoff, and append each result with its ID. On restart, skip finished IDs. The rate limit, not the model, usually sets the finish time.

## Key takeaways

-   BroadNet Inference documents chat completions and transcriptions only. There is no Batch API and no batch discount, so a bulk job is a loop of ordinary requests.
-   Finish time is items ÷ the slower of two rates: in-flight limit ÷ latency, or requests per minute. A pay-as-you-go key allows 4 in flight and 30 a minute, so one item per request tops out at 43,200 a day.
-   Write one result line per item, keyed by ID, and flush it. That line is the checkpoint: a crash costs at most the requests that were in flight.
-   A plan session runs one request at a time. Its throughput is 86,400 ÷ your measured seconds per item, per day, per session.
-   If the work can wait a day and a 50% batch discount matters, OpenAI's and Anthropic's Batch APIs may be the cheaper tool. Price both.

**On this page**

1.  Does BroadNet Inference have a Batch API?
2.  How long will the job take?
3.  What does a resumable runner look like?
4.  Why a worker pool and not asyncio.gather?
5.  How do retries stay safe?
6.  How does checkpointing work?
7.  What does it cost: pay-as-you-go or a plan session?
8.  Where BroadNet Inference fits

Some LLM jobs aren’t chats. You have 50,000 support tickets to label, 200,000 product descriptions to translate, or a million reviews to score. Providers like OpenAI and Anthropic sell a [Batch API](https://broadrouter.com/glossary/batch-api/) for this. Many OpenAI-compatible APIs, BroadNet Inference among them, don’t. This post shows how to run the job yourself: work out how long it will take, write a runner that respects the limits, and make it safe to stop and restart.

BroadRouter is BroadNet’s own site, so the worked numbers use BroadNet’s published limits. The method works on any chat-completions API; swap in your provider’s numbers.

## Does BroadNet Inference have a Batch API?

No. The [API reference](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-batch-processing-without-batch-api&utm_content=body), checked on 15 November 2026, documents `POST /v1/chat/completions`, `POST /v1/audio/transcriptions` and `GET /v1/models` (a model list), plus account endpoints under `/portal/`. There is no batch or file-upload endpoint and no batch discount. A bulk job is a loop of normal requests.

Here is what a hosted Batch API gives you, so you know what you are rebuilding:

|  | OpenAI Batch API | Anthropic Message Batches | Your own runner |
| --- | --- | --- | --- |
| Price | 50% off synchronous | 50% off | Normal price |
| Turnaround | Within 24 hours | Most within 1 hour; expires after 24 | Set by your rate limits |
| Size per batch | 50,000 requests or 200 MB | 100,000 requests or 256 MB | Any size |
| Rate limits | Separate from per-model limits | Apply to queued requests | Your key’s limits |
| Result order | May differ; match on `custom_id` | Match on `custom_id` | You key results by ID |

Two lessons carry over. Both providers make you tag each request with an ID, because results don’t come back in order. Neither promises an answer in seconds. Your runner should work the same way: IDs on everything, and a plan measured in hours.

If a 50% discount matters more than turnaround, and the models you need are there, a hosted Batch API may be the cheaper tool. Price both with the [LLM API cost calculator](https://broadrouter.com/tools/llm-api-cost-calculator/) before you build.

## How long will the job take?

Work this out before you write code. Two limits cap throughput, and the slower one wins:

-   **In-flight limit ÷ latency.** With 4 requests in flight and 2 seconds per request, you can finish 2 a second.
-   **Requests per minute.** At 30 a minute, you can start 0.5 a second, however fast the model is.

A BroadNet [pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/) key allows 4 requests in flight and 30 a minute. At the values in the API reference when we checked (30 a minute and 4 in flight, 2026-11-15), the [rate limit](https://broadrouter.com/glossary/rate-limit/) binds unless each request takes more than 8 seconds, and the table below uses 30 a minute:

| Items | One item per request | 20 items per request |
| --- | --- | --- |
| 10,000 | 5.6 hours | 17 minutes |
| 100,000 | 2.3 days | 2.8 hours |
| 1,000,000 | 23 days | 28 hours |

At 30 requests a minute, one key does 1,800 requests an hour, or 43,200 a day. Retries count too: every 429 you trigger spends a slot.

The second column is the biggest lever. Short items, such as reviews or ticket subjects, can go 10 to 50 per request as a numbered list, with a JSON array back. Check that the array has one entry per input and that each carries its number. The [Arabic text classification post](https://broadrouter.com/blog/arabic-text-classification-llm/) shows the pattern. Long items won’t pack: each request still has to fit the model’s [context window](https://broadrouter.com/glossary/context-window/), and quality can drop as the list grows. Test packing on 200 items before you trust it.

Don’t spread one job across several keys to beat a per-key limit without asking. BroadNet’s [Acceptable Use Policy](https://www.broadnet.ai/aup) forbids circumventing rate, concurrency or budget limits. If you need more, ask BroadNet, or use plan sessions, below. The [batch job planner](https://broadrouter.com/tools/llm-batch-job-planner/) does this arithmetic for your item count, latency and limits.

## What does a resumable runner look like?

This runner reads `items.jsonl` (one `{"id", "text"}` object per line) and appends one result line per item to `results.jsonl`. It needs only the OpenAI Python SDK.

```python
import asyncio, json, os, random, time
from openai import AsyncOpenAI, APIStatusError, APIConnectionError, APITimeoutError

client = AsyncOpenAI(base_url=os.environ.get("BASE_URL", "https://inference.broadnet.ai/v1"),
                     api_key=os.environ["BROADNET_API_KEY"],
                     max_retries=0,        # we retry ourselves, with a cap we can see
                     timeout=120)
MODEL = os.environ.get("MODEL", "broadnet/deepseek-v4.1-flash")
IN_FLIGHT = int(os.environ.get("IN_FLIGHT", "4"))     # PAYG key: 4 in flight. Plan: 1 per session.
RPM = int(os.environ.get("RPM", "30"))                # PAYG key: 30 requests a minute
MAX_ATTEMPTS = 6

class Pacer:
    """Spaces request starts at least 60/RPM seconds apart, across all workers."""
    def __init__(self, rpm):
        self.gap, self.next, self.lock = 60 / rpm, 0.0, asyncio.Lock()
    async def wait(self):
        async with self.lock:
            now = time.monotonic()
            delay = max(0.0, self.next - now)
            self.next = max(now, self.next) + self.gap
        await asyncio.sleep(delay)

def retryable(e):
    if isinstance(e, (APIConnectionError, APITimeoutError)):
        return True
    return isinstance(e, APIStatusError) and e.status_code in (429, 500, 502, 503, 504)

async def call(item, pacer):
    for attempt in range(MAX_ATTEMPTS):
        await pacer.wait()                          # every attempt counts against the rate limit
        try:
            r = await client.chat.completions.create(
                model=MODEL, max_tokens=1500, reasoning_effort="low",
                messages=[{"role": "system", "content": "Classify the review as positive, negative or mixed. Reply with one word."},
                          {"role": "user", "content": item["text"]}])
            c = r.choices[0]
            return {"id": item["id"], "ok": True, "finish_reason": c.finish_reason,
                    "output": c.message.content, "usage": r.usage.model_dump() if r.usage else None}
        except Exception as e:
            if not retryable(e) or attempt == MAX_ATTEMPTS - 1:
                return {"id": item["id"], "ok": False, "error": repr(e)[:300]}
            await asyncio.sleep(random.uniform(0, min(60, 2 ** attempt)))   # full jitter, capped

async def run(in_path, out_path):
    done = set()
    if os.path.exists(out_path):                    # resume: skip ids that already succeeded
        with open(out_path) as f:
            done = {json.loads(l)["id"] for l in f if l.strip() and json.loads(l).get("ok")}
    todo = [it for it in map(json.loads, open(in_path)) if it["id"] not in done]
    print(f"{len(done)} done, {len(todo)} to go")
    pacer, queue = Pacer(RPM), asyncio.Queue()
    for it in todo:
        queue.put_nowait(it)
    with open(out_path, "a") as out:
        async def worker():
            while True:
                try:
                    it = queue.get_nowait()
                except asyncio.QueueEmpty:
                    return
                res = await call(it, pacer)
                out.write(json.dumps(res, ensure_ascii=False) + "\n")
                out.flush()                         # one line per finished item = the checkpoint
        await asyncio.gather(*(worker() for _ in range(IN_FLIGHT)))

if __name__ == "__main__":
    import sys
    asyncio.run(run(sys.argv[1], sys.argv[2]))
```

Run it with `python batch_run.py items.jsonl results.jsonl`. We tested this exact file against a local mock server that returned random 429s (8%) and 500s (5%) and refused anything over 4 in flight or 30 starts a minute. 25 items finished in 55 seconds with 28 requests (3 retries), and never more than 30 starts in any 60-second window. With the mock’s limit raised to 300 a minute to save time, a 120-item run killed after 10 seconds resumed from 39 saved results and finished with 120 unique IDs and no duplicates.

## Why a worker pool and not asyncio.gather?

The [429 guide](https://broadrouter.com/blog/handle-llm-api-429-rate-limits/) uses a semaphore around `asyncio.gather`. That is fine for a few hundred calls. For a large job, a fixed pool of workers pulling from a queue is better:

-   **Memory.** `gather` creates one task per item up front. A pool holds only the items in flight.
-   **Concurrency is the pool size.** Four workers can never have five requests open.
-   **Results stream to disk** as they finish, in whatever order, keyed by ID.

The `Pacer` handles the second limit. A semaphore alone stops you exceeding 4 in flight, but if requests take one second, four workers would start 240 a minute and hit 429 on most of them. The pacer spaces starts 2 seconds apart, across all workers, retries included.

## How do retries stay safe?

Retry only what can succeed on a second try:

| Response | Action |
| --- | --- |
| 429 | Back off and retry; BroadNet’s docs say “use response details and bounded backoff” |
| 500, 502, 503, 504, timeouts, dropped connections | Retry with bounded exponential backoff |
| 400, 422 | Fix the request; retrying sends the same error |
| 401, 402, 403 | Stop the job; key, balance or policy needs attention |
| 413 | Shorten the item |

The runner turns off the SDK’s own retries (`max_retries=0`) so there is exactly one retry policy, with a visible cap. Backoff is [exponential](https://broadrouter.com/glossary/exponential-backoff/) with full jitter, capped at 60 seconds, so workers that failed together don’t retry together. After six attempts the item is written as `"ok": false` and the job moves on. One bad item never stops a million.

A model call has no side effects, so delivery here is at least once: if the process dies mid-request, that item is sent again on restart, and you pay for it twice. That is the only waste. If the job also writes to another system, such as tickets or emails, make that write [idempotent](https://broadrouter.com/glossary/idempotency-key/) by item ID and do it in a separate pass over `results.jsonl`. The [long-running agents post](https://broadrouter.com/blog/long-running-agents-timeouts-retries/) covers idempotent side effects in depth.

## How does checkpointing work?

The output file is the checkpoint. Each finished item is one flushed line, so after a crash, a deploy or Ctrl-C, the next run reads the file, collects IDs with `"ok": true` and queues only the rest. Items that failed earlier are retried on the next run, and their new line is appended. When you read results, take the last line per ID.

Three habits make this hold up over days:

1.  **Stable IDs.** Use a database key or a hash of the input, never a line number that shifts when the input file changes.
2.  **Check `finish_reason`.** A reply cut off at `max_tokens` is `"ok": true` but useless. DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before they answer, and [reasoning counts toward max\_tokens](https://broadrouter.com/guides/reasoning-tokens-empty-response/). The runner sets `reasoning_effort="low"`, which BroadNet documents, and leaves 1,500 tokens.
3.  **Log usage per line.** Summing `usage` from `results.jsonl` gives the real token bill to compare with your estimate.

## What does it cost: pay-as-you-go or a plan session?

On pay-as-you-go, cost is tokens. Take 100,000 reviews, a 150-token system prompt, a 150-token review and 100 output tokens with low reasoning. That is 30 million input and 10 million output tokens. On [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) it costs 30 × $0.175 + 10 × $0.52. Keep the system prompt first and byte-identical, and its 15 million tokens can bill at the [cached input](https://broadrouter.com/blog/cached-input-tokens-explained/) rate of $0.0055 per 1M where the cache hits. Your `usage` lines show how often it did.

On a plan, tokens are free and time is the cost. A DeepSeek V4.1 Flash plan session “runs one request at a time”, so set `IN_FLIGHT=1` per session. Throughput per session is 86,400 ÷ seconds per item, per day. BroadNet quotes typical speed as about 100 tokens per second, but your latency depends on prompt length and reasoning, so measure it on 50 items. If you measure 3 seconds an item, one session does 28,800 a day:

| Job | Sessions | Time at 3 s/item | Plan |
| --- | --- | --- | --- |
| 100,000 items | 1 | About 3.5 days | One Orbit week, $15 per session |
| 100,000 items | 4 | About 21 hours | Four Orbit weeks, one per session; or about 21 Meteor hours per session at $1 each |
| 1,000,000 items | 6 | About 6 days | Six Orbit weeks |

Plans are delivered “best effort under fair use”, and BroadNet doesn’t publish a per-minute request limit for plan keys; your account shows the limits that apply. Run `RPM` from those, not from guesswork. The [session planner guide](https://broadrouter.com/guides/unlimited-llm-api-sessions/) explains how to split sessions across keys.

## Where BroadNet Inference fits

BroadNet Inference has no Batch API, so it suits bulk jobs when you would rather control the runner yourself and pay normal per-token prices, or fix the cost with DeepSeek V4.1 Flash plan sessions. For small jobs on a budget, [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/) at $0.10 input and $0.22 output per 1M runs on the $5 starting credit. Prompts and outputs are not retained after each request. If your job can wait a day and the models you need are on a provider with a 50% batch discount, that may cost less; compare before you commit. Plans and limits are on [inference.broadnet.ai/plans](https://inference.broadnet.ai/plans?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-batch-processing-without-batch-api&utm_content=body).

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Batch API](https://broadrouter.com/glossary/batch-api/)

A batch API is an asynchronous LLM endpoint that accepts many requests at once, usually as a file, processes them within a set window instead of immediately, and returns the results together. Providers such as OpenAI and Anthropic price batch requests at a discount to their standard APIs.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Rate limit](https://broadrouter.com/glossary/rate-limit/)

A rate limit is a cap an API places on how much a client can use it in a time window, such as requests per minute or tokens per minute. Requests over the limit are refused, usually with HTTP 429 Too Many Requests, until usage falls back under it.

[Concurrency](https://broadrouter.com/glossary/concurrency/)

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

[Latency](https://broadrouter.com/glossary/latency/)

Latency, for a language model API, is the time a request takes from being sent to the final output token being received. It is the sum of time to first token and generation time, and is usually reported as percentiles such as p50 and p95 rather than as an average.

[Exponential backoff](https://broadrouter.com/glossary/exponential-backoff/)

Exponential backoff is a retry strategy in which a client waits longer after each failed attempt, typically doubling the delay up to a cap, before trying again. Adding random jitter to each wait spreads out retries from many clients so they do not hit the server at the same moment.

[Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/)

Input tokens are the tokens a language model reads in a request, including the system prompt, conversation history, tool definitions and images. Output tokens are the tokens it generates in reply, including any reasoning. LLM APIs usually price them separately, and output usually costs more.

## Questions

### Does BroadNet Inference have a Batch API?

No. Its API reference, checked on 15 November 2026, documents POST /v1/chat/completions, POST /v1/audio/transcriptions and GET /v1/models, with no batch or file-upload endpoint and no batch discount. Run bulk jobs as ordinary requests within your key's limits.

### How do I process 100,000 prompts with an LLM API?

Put the items in a JSONL file with an ID each. Run a fixed number of workers equal to the key's in-flight limit, pace request starts under the per-minute limit, retry 429 and 5xx with capped backoff, and append each result to an output file keyed by ID so the job can resume.

### How long will a batch of a million requests take?

Divide the items by your real rate. At 30 requests a minute and one item per request, one key does 43,200 a day, so a million takes about 23 days. Packing 20 short items per request cuts that to about 28 hours, if the model handles the longer prompt well.

### Should I use asyncio.gather or a worker pool for batch LLM calls?

For tens of thousands of items, use a fixed pool of workers pulling from a queue. gather with a semaphore works for small jobs, but it creates every task up front and makes resuming and progress tracking harder.

### Can I use several API keys to go faster?

Check first. BroadNet's Acceptable Use Policy forbids circumventing rate or concurrency limits, so ask BroadNet before you split one job across keys to get past a per-key limit. Packing more items per request and plan sessions are the documented ways to add throughput.

### Is a flat plan good for batch jobs?

It can be, for DeepSeek V4.1 Flash. A session runs one request at a time with no token meter, so cost is fixed and the question is time. Measure seconds per item, then buy enough sessions to finish inside the period.

## Keep reading

-   [How to handle LLM API 429 errors: backoff, jitter and limiters](https://broadrouter.com/blog/handle-llm-api-429-rate-limits/)
-   [Long-running agents: timeouts, retries and checkpoints that hold](https://broadrouter.com/blog/long-running-agents-timeouts-retries/)
-   [Arabic text classification with an LLM: sentiment, intent, triage](https://broadrouter.com/blog/arabic-text-classification-llm/)
-   [How to run parallel coding agents on a flat-rate LLM API](https://broadrouter.com/blog/run-parallel-coding-agents-flat-rate/)
-   [A weekly LLM plan for short projects: Orbit vs hour vs month](https://broadrouter.com/blog/weekly-llm-plan-short-projects/)
-   [How to estimate LLM API cost before you launch a feature](https://broadrouter.com/blog/estimate-llm-api-cost-before-launch/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)
-   [Guide: How the unlimited LLM API plans work for agents](https://broadrouter.com/guides/unlimited-llm-api-sessions/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-batch-processing-without-batch-api&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference API reference: endpoints, pay-as-you-go limits (30 a minute, 4 in flight), hourly key limits, 429 and 5xx retry guidance, reasoning\_effort (checked 2026-11-15)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-batch-processing-without-batch-api&utm_content=sources)
-   [BroadNet Inference plans page: one request at a time per session, typical speed, plan prices (checked 2026-11-15)](https://inference.broadnet.ai/plans?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-batch-processing-without-batch-api&utm_content=sources)
-   [BroadNet Acceptable Use Policy, sections 3 and 4: no circumventing rate or concurrency limits; plans best effort under fair use (checked 2026-11-15)](https://www.broadnet.ai/aup)
-   [OpenAI Batch API guide: 50% discount, 24h window, 50,000 requests or 200 MB per batch, output order may differ, custom\_id (checked 2026-11-15)](https://developers.openai.com/api/docs/guides/batch)
-   [Anthropic Message Batches API docs: 50% discount, 100,000 requests or 256 MB, most batches within 1 hour, expiry after 24 hours (checked 2026-11-15)](https://platform.claude.com/docs/en/build-with-claude/batch-processing)
