Serverless does not mean there are no servers. It means you never see them, and you stop paying when you stop sending requests.
How it works
AWS’s SageMaker guide gives a clear reference version. Serverless endpoints “automatically launch compute resources and scale them in and out depending on traffic”, and when there are no requests the service “scales your endpoint down to 0”. You pay per use rather than per instance.
The cost of scaling to zero is the cold start. If an endpoint has been idle and a request arrives, “it can take some time for your endpoint to spin up the compute resources”. For large language models, loading tens of gigabytes of weights makes that delay larger. The usual fix is to pay to keep some capacity warm; AWS calls this provisioned concurrency and bills it by memory, duration and concurrency.
Features vary by platform. AWS’s own list of exclusions for SageMaker Serverless Inference includes GPUs, so LLM-sized serverless hosting usually comes from other platforms. Hugging Face describes its Inference Providers as a way to use “serverless inference of models” across several providers with one token.
Why it matters
| Serverless endpoint | Rented GPU (GPU-hour) | Shared per-token API | |
|---|---|---|---|
| Pay when idle | No (unless kept warm) | Yes | No |
| Cold starts | Possible | No, once running | Rare for listed models |
| Your own model | Yes | Yes | Only what the provider lists |
| Scaling work | Platform | You | Provider |
Serverless suits spiky or low traffic with your own model. A steady, heavy load usually costs less on reserved hardware. Cold starts show up as a long latency tail, not in the average, so check p95.
On BroadNet Inference
BroadNet does not describe its API as serverless, and its docs say nothing about cold starts. In practice, its pay-as-you-go keys share the serverless traits that matter to a caller: no instance to choose, nothing to scale, and billing per token or per audio minute. The limits are per key, 30 requests per minute and 4 in flight, rather than per machine. You can only call the models BroadNet lists, such as Qwen3.8 27B; private endpoints and other models are on request. A first request with curl takes a minute to try.