---
title: "What is serverless inference? Pay-per-use model hosting"
description: "Serverless inference runs a model behind an endpoint that scales with traffic, even to zero, and bills only for use. Cold starts, pricing, and how it compares."
url: https://broadrouter.com/glossary/serverless-inference/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Serverless inference

Updated 2026-11-16

Serverless inference is a way to run machine learning models in which the platform starts, scales and stops the compute behind an endpoint automatically, often down to zero when idle, and bills for usage rather than for reserved machines. The trade-off is possible cold-start delay.

Also called: serverless LLM, serverless model endpoint, scale to zero, serverless GPU inference

Serverless does not mean there are no servers. It means you never see them, and you stop paying when you stop sending requests.

## How it works

AWS’s SageMaker guide gives a clear reference version. Serverless endpoints “automatically launch compute resources and scale them in and out depending on traffic”, and when there are no requests the service “scales your endpoint down to 0”. You pay per use rather than per instance.

The cost of scaling to zero is the **cold start**. If an endpoint has been idle and a request arrives, “it can take some time for your endpoint to spin up the compute resources”. For large language models, loading tens of gigabytes of weights makes that delay larger. The usual fix is to pay to keep some capacity warm; AWS calls this provisioned concurrency and bills it by memory, duration and concurrency.

Features vary by platform. AWS’s own list of exclusions for SageMaker Serverless Inference includes GPUs, so LLM-sized serverless hosting usually comes from other platforms. Hugging Face describes its Inference Providers as a way to use “serverless inference of models” across several providers with one token.

## Why it matters

|  | Serverless endpoint | Rented GPU ([GPU-hour](https://broadrouter.com/glossary/gpu-hour/)) | Shared per-token API |
| --- | --- | --- | --- |
| Pay when idle | No (unless kept warm) | Yes | No |
| Cold starts | Possible | No, once running | Rare for listed models |
| Your own model | Yes | Yes | Only what the provider lists |
| Scaling work | Platform | You | Provider |

Serverless suits spiky or low traffic with your own model. A steady, heavy load usually costs less on reserved hardware. Cold starts show up as a long [latency](https://broadrouter.com/glossary/latency/) tail, not in the average, so check p95.

## On BroadNet Inference

BroadNet does not describe its API as serverless, and its docs say nothing about cold starts. In practice, its [pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/) keys share the serverless traits that matter to a caller: no instance to choose, nothing to scale, and billing per token or per audio minute. The limits are per key, 30 requests per minute and 4 in flight, rather than per machine. You can only call the models BroadNet lists, such as [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/); private endpoints and other models are on request. A [first request with curl](https://broadrouter.com/guides/curl-quickstart/) takes a minute to try.

## Questions

### What is a cold start in serverless inference?

The delay when an endpoint that has scaled down receives a request and must start compute and load the model first. AWS's SageMaker docs describe it and offer provisioned concurrency to keep capacity warm for a fee.

### Is a per-token LLM API serverless?

From the caller's side, mostly yes. You send requests, pay per token and manage no servers. The difference is that a shared API keeps popular models loaded for many customers, so you do not usually wait for your own model to start.

### Is serverless inference cheaper than a dedicated GPU?

For spiky or low traffic, usually, because you stop paying when idle. For steady heavy load, a reserved GPU often costs less per request. Compare your traffic pattern against both prices.

## See also

-   [Inference provider](https://broadrouter.com/glossary/inference-provider/) (term)
-   [GPU-hour](https://broadrouter.com/glossary/gpu-hour/) (term)
-   [Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/) (term)
-   [Latency](https://broadrouter.com/glossary/latency/) (term)
-   [Concurrency](https://broadrouter.com/glossary/concurrency/) (term)
-   [First request to BroadNet Inference with curl](https://broadrouter.com/guides/curl-quickstart/) (guide)
-   [Self-hosting an LLM vs an API: the real cost and break-even](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) (article)
-   [LLM API pricing explained: input, cached, output and reasoning](https://broadrouter.com/blog/llm-api-pricing-explained/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-serverless-inference&utm_content=page-cta) ·

**Sources**

-   [AWS, Amazon SageMaker Serverless Inference developer guide (checked 2026-11-16)](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html)
-   [Hugging Face, Inference Providers documentation (serverless inference) (checked 2026-11-16)](https://huggingface.co/docs/inference-providers/index)
-   [BroadNet Inference API reference, limits (checked 2026-11-16)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-serverless-inference&utm_content=sources#limits)
