BroadRouter
Get started

Serverless inference

Updated

Serverless inference is a way to run machine learning models in which the platform starts, scales and stops the compute behind an endpoint automatically, often down to zero when idle, and bills for usage rather than for reserved machines. The trade-off is possible cold-start delay.

Also called: serverless LLM, serverless model endpoint, scale to zero, serverless GPU inference

Serverless does not mean there are no servers. It means you never see them, and you stop paying when you stop sending requests.

How it works

AWS’s SageMaker guide gives a clear reference version. Serverless endpoints “automatically launch compute resources and scale them in and out depending on traffic”, and when there are no requests the service “scales your endpoint down to 0”. You pay per use rather than per instance.

The cost of scaling to zero is the cold start. If an endpoint has been idle and a request arrives, “it can take some time for your endpoint to spin up the compute resources”. For large language models, loading tens of gigabytes of weights makes that delay larger. The usual fix is to pay to keep some capacity warm; AWS calls this provisioned concurrency and bills it by memory, duration and concurrency.

Features vary by platform. AWS’s own list of exclusions for SageMaker Serverless Inference includes GPUs, so LLM-sized serverless hosting usually comes from other platforms. Hugging Face describes its Inference Providers as a way to use “serverless inference of models” across several providers with one token.

Why it matters

Serverless endpoint Rented GPU (GPU-hour) Shared per-token API
Pay when idle No (unless kept warm) Yes No
Cold starts Possible No, once running Rare for listed models
Your own model Yes Yes Only what the provider lists
Scaling work Platform You Provider

Serverless suits spiky or low traffic with your own model. A steady, heavy load usually costs less on reserved hardware. Cold starts show up as a long latency tail, not in the average, so check p95.

On BroadNet Inference

BroadNet does not describe its API as serverless, and its docs say nothing about cold starts. In practice, its pay-as-you-go keys share the serverless traits that matter to a caller: no instance to choose, nothing to scale, and billing per token or per audio minute. The limits are per key, 30 requests per minute and 4 in flight, rather than per machine. You can only call the models BroadNet lists, such as Qwen3.8 27B; private endpoints and other models are on request. A first request with curl takes a minute to try.

Questions

What is a cold start in serverless inference?

The delay when an endpoint that has scaled down receives a request and must start compute and load the model first. AWS's SageMaker docs describe it and offer provisioned concurrency to keep capacity warm for a fee.

Is a per-token LLM API serverless?

From the caller's side, mostly yes. You send requests, pay per token and manage no servers. The difference is that a shared API keeps popular models loaded for many customers, so you do not usually wait for your own model to start.

Is serverless inference cheaper than a dedicated GPU?

For spiky or low traffic, usually, because you stop paying when idle. For steady heavy load, a reserved GPU often costs less per request. Compare your traffic pattern against both prices.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity