BroadRouter
Get started

Batch API

Updated

A batch API is an asynchronous LLM endpoint that accepts many requests at once, usually as a file, processes them within a set window instead of immediately, and returns the results together. Providers such as OpenAI and Anthropic price batch requests at a discount to their standard APIs.

Also called: batch processing API, Message Batches API, OpenAI Batch API, batch inference, async batch requests

Many LLM jobs do not need an answer now. A batch API trades speed of reply for a lower price and higher volume.

How it works

You prepare many requests, submit them together, and collect the results later. The two best-documented versions:

OpenAI Batch API Anthropic Message Batches API
Input A .jsonl file, one request per line A list of Messages requests
Price “50% cost discount compared to synchronous APIs” “50 percent discount on all usage”
Turnaround “Each batch completes within 24 hours (and often more quickly)” Most within 1 hour; expires if not done within 24 hours
Size limit 50,000 requests, 200 MB input file 100,000 requests or 256 MB per batch
Rate limits Separate from per-model limits Apply to queued batch requests too

With OpenAI you upload the file, create a batch with completion_window: "24h", poll its status and download the output file. Both APIs require a unique custom_id per request, because results can come back in a different order from the input. Anthropic keeps results for 29 days, and its batch and prompt caching discounts can stack.

Why it matters

Good fits are evaluations, classifying a large dataset, bulk summaries and back-filling. OpenAI’s guide names running evaluations and classifying large datasets as typical uses. Bad fits are anything a user is waiting for.

The savings are real but bounded. A batch halves the token price on those providers; it does not reduce how many input and output tokens you send. Trimming prompts still matters.

A batch also sidesteps the rate limit problem. Instead of managing 429s and backoff yourself, the provider schedules the work.

On BroadNet Inference

BroadNet’s docs do not list a batch API, a files endpoint or a batch discount. Bulk work on BroadNet means synchronous requests run in parallel within your key’s limits: 30 requests per minute and 4 in flight on a pay-as-you-go key. Handle 429 and 5xx responses with bounded exponential backoff, as the errors and retries guide shows.

For long jobs on DeepSeek V4.1 Flash, a flat plan session runs one request at a time with no token meter, so cost is fixed by time instead of volume. Use the flat-rate vs per-token calculator to compare the two for your job size, and the LLM batch job planner for the run time under your limits and the sessions a deadline needs.

Questions

How much cheaper is a batch API?

OpenAI and Anthropic both document a 50% discount compared with their standard APIs. Anthropic's discount applies to input, output and special tokens.

How long does a batch take?

OpenAI states each batch completes within 24 hours, often sooner. Anthropic says most batches finish within 1 hour and expire if processing is not done within 24 hours.

Does BroadNet have a batch API?

BroadNet's docs do not list one. You can run bulk jobs by sending requests in parallel within your key's limits, or on a flat plan for DeepSeek V4.1 Flash.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity