Many LLM jobs do not need an answer now. A batch API trades speed of reply for a lower price and higher volume.
How it works
You prepare many requests, submit them together, and collect the results later. The two best-documented versions:
| OpenAI Batch API | Anthropic Message Batches API | |
|---|---|---|
| Input | A .jsonl file, one request per line |
A list of Messages requests |
| Price | “50% cost discount compared to synchronous APIs” | “50 percent discount on all usage” |
| Turnaround | “Each batch completes within 24 hours (and often more quickly)” | Most within 1 hour; expires if not done within 24 hours |
| Size limit | 50,000 requests, 200 MB input file | 100,000 requests or 256 MB per batch |
| Rate limits | Separate from per-model limits | Apply to queued batch requests too |
With OpenAI you upload the file, create a batch with completion_window: "24h", poll its status and download the output file. Both APIs require a unique custom_id per request, because results can come back in a different order from the input. Anthropic keeps results for 29 days, and its batch and prompt caching discounts can stack.
Why it matters
Good fits are evaluations, classifying a large dataset, bulk summaries and back-filling. OpenAI’s guide names running evaluations and classifying large datasets as typical uses. Bad fits are anything a user is waiting for.
The savings are real but bounded. A batch halves the token price on those providers; it does not reduce how many input and output tokens you send. Trimming prompts still matters.
A batch also sidesteps the rate limit problem. Instead of managing 429s and backoff yourself, the provider schedules the work.
On BroadNet Inference
BroadNet’s docs do not list a batch API, a files endpoint or a batch discount. Bulk work on BroadNet means synchronous requests run in parallel within your key’s limits: 30 requests per minute and 4 in flight on a pay-as-you-go key. Handle 429 and 5xx responses with bounded exponential backoff, as the errors and retries guide shows.
For long jobs on DeepSeek V4.1 Flash, a flat plan session runs one request at a time with no token meter, so cost is fixed by time instead of volume. Use the flat-rate vs per-token calculator to compare the two for your job size, and the LLM batch job planner for the run time under your limits and the sessions a deadline needs.