---
title: "LLM API glossary: tokens, context, routers and pricing"
description: "Plain definitions of the terms you meet when calling an LLM or speech-to-text API: tokens, context windows, cached input, rate limits and zero data retention."
url: https://broadrouter.com/glossary/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# LLM API glossary

Updated 2026-11-16

Short, sourced definitions of the terms developers meet when they call an LLM or speech-to-text API: what tokens and context windows are, how cached input and flat-rate plans change the bill, and what rate limits, routers and zero data retention mean in practice.

[AI agent](https://broadrouter.com/glossary/agent/)[AI coding agent](https://broadrouter.com/glossary/coding-agent/)[API key](https://broadrouter.com/glossary/api-key/)[Arabic diacritics (tashkeel)](https://broadrouter.com/glossary/arabic-diacritics/)[Arabic dialect identification](https://broadrouter.com/glossary/arabic-dialect-identification/)[Arabic text normalization](https://broadrouter.com/glossary/arabic-text-normalization/)[Arabizi](https://broadrouter.com/glossary/arabizi/)[AWQ (activation-aware weight quantization)](https://broadrouter.com/glossary/awq/)[Base URL](https://broadrouter.com/glossary/base-url/)[Batch API](https://broadrouter.com/glossary/batch-api/)[BM25](https://broadrouter.com/glossary/bm25/)[Byte pair encoding (BPE)](https://broadrouter.com/glossary/bpe/)[Cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/)[Chain-of-thought (CoT)](https://broadrouter.com/glossary/chain-of-thought/)[Character error rate (CER)](https://broadrouter.com/glossary/character-error-rate/)[Chat Completions API](https://broadrouter.com/glossary/chat-completions-api/)[Chat template](https://broadrouter.com/glossary/chat-template/)[Chunking (RAG)](https://broadrouter.com/glossary/chunking/)[Code-switching](https://broadrouter.com/glossary/code-switching/)[Concurrency](https://broadrouter.com/glossary/concurrency/)[Context compaction](https://broadrouter.com/glossary/context-compaction/)[Context engineering](https://broadrouter.com/glossary/context-engineering/)[Context rot](https://broadrouter.com/glossary/context-rot/)[Context window](https://broadrouter.com/glossary/context-window/)[Data processing agreement (DPA)](https://broadrouter.com/glossary/dpa/)[Data residency](https://broadrouter.com/glossary/data-residency/)[Dialectal Arabic](https://broadrouter.com/glossary/dialectal-arabic/)[Distillation (knowledge distillation)](https://broadrouter.com/glossary/distillation/)[Embeddings](https://broadrouter.com/glossary/embeddings/)[Eval (LLM evaluation)](https://broadrouter.com/glossary/eval/)[Exponential backoff](https://broadrouter.com/glossary/exponential-backoff/)[Few-shot prompting](https://broadrouter.com/glossary/few-shot-prompting/)[Fine-tuning](https://broadrouter.com/glossary/fine-tuning/)[finish\_reason](https://broadrouter.com/glossary/finish-reason/)[Flat-rate LLM plan](https://broadrouter.com/glossary/flat-rate-llm-plan/)[FP8](https://broadrouter.com/glossary/fp8/)[Frequency and presence penalty](https://broadrouter.com/glossary/frequency-and-presence-penalty/)[GDPR](https://broadrouter.com/glossary/gdpr/)[GGUF](https://broadrouter.com/glossary/gguf/)[GPU-hour](https://broadrouter.com/glossary/gpu-hour/)[Grounding](https://broadrouter.com/glossary/grounding/)[Guardrails](https://broadrouter.com/glossary/guardrails/)[Hallucination](https://broadrouter.com/glossary/hallucination/)[Hamza](https://broadrouter.com/glossary/hamza/)[Hybrid search](https://broadrouter.com/glossary/hybrid-search/)[Idempotency key](https://broadrouter.com/glossary/idempotency-key/)[Image tokens](https://broadrouter.com/glossary/image-tokens/)[Inference provider](https://broadrouter.com/glossary/inference-provider/)[Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/)[INT4](https://broadrouter.com/glossary/int4/)[IP allow list](https://broadrouter.com/glossary/ip-allow-list/)[Jailbreak (LLM)](https://broadrouter.com/glossary/jailbreak/)[JSON mode](https://broadrouter.com/glossary/json-mode/)[JSON Schema](https://broadrouter.com/glossary/json-schema/)[KV cache](https://broadrouter.com/glossary/kv-cache/)[Latency](https://broadrouter.com/glossary/latency/)[LLM benchmark](https://broadrouter.com/glossary/llm-benchmark/)[LLM router](https://broadrouter.com/glossary/llm-router/)[LLM-as-a-judge](https://broadrouter.com/glossary/llm-as-judge/)[Logprobs](https://broadrouter.com/glossary/logprobs/)[LoRA (low-rank adaptation)](https://broadrouter.com/glossary/lora/)[Lost in the middle](https://broadrouter.com/glossary/lost-in-the-middle/)[max\_tokens](https://broadrouter.com/glossary/max-tokens/)[MCP (Model Context Protocol)](https://broadrouter.com/glossary/mcp/)[Mixture of experts (MoE)](https://broadrouter.com/glossary/mixture-of-experts/)[Model deprecation](https://broadrouter.com/glossary/model-deprecation/)[Model fallback](https://broadrouter.com/glossary/model-fallback/)[Model license](https://broadrouter.com/glossary/model-license/)[Modern Standard Arabic (MSA)](https://broadrouter.com/glossary/msa/)[Multi-agent system](https://broadrouter.com/glossary/multi-agent-system/)[Multimodal AI](https://broadrouter.com/glossary/multimodal/)[Named entity recognition (NER)](https://broadrouter.com/glossary/named-entity-recognition/)[Ogg Opus](https://broadrouter.com/glossary/ogg-opus/)[Open-weight model](https://broadrouter.com/glossary/open-weight-model/)[OpenAI-compatible API](https://broadrouter.com/glossary/openai-compatible-api/)[Optical character recognition (OCR)](https://broadrouter.com/glossary/ocr/)[Parallel tool calls](https://broadrouter.com/glossary/parallel-tool-calls/)[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)[PDPL (Saudi and UAE data protection laws)](https://broadrouter.com/glossary/pdpl/)[PII (personally identifiable information)](https://broadrouter.com/glossary/pii/)[Prefill and decode](https://broadrouter.com/glossary/prefill-and-decode/)[Prepaid credit](https://broadrouter.com/glossary/prepaid-credit/)[Prompt engineering](https://broadrouter.com/glossary/prompt-engineering/)[Prompt injection](https://broadrouter.com/glossary/prompt-injection/)[Quantization](https://broadrouter.com/glossary/quantization/)[RAG (retrieval-augmented generation)](https://broadrouter.com/glossary/rag/)[Rate limit](https://broadrouter.com/glossary/rate-limit/)[ReAct (reason and act)](https://broadrouter.com/glossary/react-agent/)[Reasoning tokens](https://broadrouter.com/glossary/reasoning-tokens/)[Request timeout](https://broadrouter.com/glossary/request-timeout/)[Reranking](https://broadrouter.com/glossary/reranking/)[Right-to-left (RTL) text](https://broadrouter.com/glossary/rtl-text/)[Sample rate](https://broadrouter.com/glossary/sample-rate/)[Seed parameter](https://broadrouter.com/glossary/seed-parameter/)[Server-sent events (SSE)](https://broadrouter.com/glossary/sse/)[Serverless inference](https://broadrouter.com/glossary/serverless-inference/)[SLA (service level agreement)](https://broadrouter.com/glossary/sla/)[Speaker diarization](https://broadrouter.com/glossary/diarization/)[Speculative decoding](https://broadrouter.com/glossary/speculative-decoding/)[Speech-to-text](https://broadrouter.com/glossary/speech-to-text/)[Spend limit](https://broadrouter.com/glossary/spend-limit/)[SRT subtitles](https://broadrouter.com/glossary/srt-subtitles/)[Stop sequences](https://broadrouter.com/glossary/stop-sequences/)[Streaming](https://broadrouter.com/glossary/streaming/)[Structured output](https://broadrouter.com/glossary/structured-output/)[Subagent](https://broadrouter.com/glossary/subagent/)[Subprocessor](https://broadrouter.com/glossary/subprocessor/)[System prompt](https://broadrouter.com/glossary/system-prompt/)[Taa marbuta](https://broadrouter.com/glossary/taa-marbuta/)[Temperature](https://broadrouter.com/glossary/temperature/)[Text-to-speech (TTS)](https://broadrouter.com/glossary/text-to-speech/)[Throughput](https://broadrouter.com/glossary/throughput/)[Time to first token (TTFT)](https://broadrouter.com/glossary/time-to-first-token/)[Timestamps (speech-to-text)](https://broadrouter.com/glossary/timestamps-asr/)[Token](https://broadrouter.com/glossary/token/)[Tokenizer](https://broadrouter.com/glossary/tokenizer/)[Tokens per second](https://broadrouter.com/glossary/tokens-per-second/)[Tool calling](https://broadrouter.com/glossary/tool-calling/)[tool\_choice](https://broadrouter.com/glossary/tool-choice/)[Top-p (nucleus sampling)](https://broadrouter.com/glossary/top-p/)[Unicode normalization](https://broadrouter.com/glossary/unicode-normalization/)[Vector database](https://broadrouter.com/glossary/vector-database/)[Vision-language model](https://broadrouter.com/glossary/vision-language-model/)[vLLM](https://broadrouter.com/glossary/vllm/)[Vocabulary size](https://broadrouter.com/glossary/vocabulary-size/)[Voice activity detection (VAD)](https://broadrouter.com/glossary/vad/)[WAV](https://broadrouter.com/glossary/wav/)[Word error rate](https://broadrouter.com/glossary/word-error-rate/)[Zero data retention](https://broadrouter.com/glossary/zero-data-retention/)

## APIs and integration

[AI agent](https://broadrouter.com/glossary/agent/)

An AI agent is a system in which a language model directs its own steps toward a goal: it decides which tools to call, reads the results and repeats until it produces a final answer or hits a limit. A fixed script that calls a model at set points is usually called a workflow instead.

[AI coding agent](https://broadrouter.com/glossary/coding-agent/)

An AI coding agent is a software tool in which a language model works on a codebase in a loop: it reads files, edits them, runs commands such as tests, reads the output and decides the next step until the task is done or a limit is reached.

[Base URL](https://broadrouter.com/glossary/base-url/)

A base URL is the fixed root address of an API, such as https://api.example.com/v1, that a client puts in front of every endpoint path, such as /chat/completions. In OpenAI-compatible SDKs, changing the base URL, together with the API key and model ID, points the same code at a different provider.

[Batch API](https://broadrouter.com/glossary/batch-api/)

A batch API is an asynchronous LLM endpoint that accepts many requests at once, usually as a file, processes them within a set window instead of immediately, and returns the results together. Providers such as OpenAI and Anthropic price batch requests at a discount to their standard APIs.

[BM25](https://broadrouter.com/glossary/bm25/)

BM25 (Okapi BM25) is a ranking function for keyword search that scores a document by how often the query terms appear in it, how rare those terms are across the collection, and how long the document is. It is the default similarity in Lucene and Elasticsearch.

[Chat Completions API](https://broadrouter.com/glossary/chat-completions-api/)

The Chat Completions API is OpenAI's HTTP endpoint, POST /v1/chat/completions, that takes a model name and a list of role-tagged messages and returns the model's next message. Its request and response shape became the common format that OpenAI-compatible providers and SDKs reuse.

[Chunking (RAG)](https://broadrouter.com/glossary/chunking/)

Chunking is the step in a retrieval pipeline that splits documents into smaller pieces, called chunks, before they are embedded and indexed. Chunk size sets how much text each piece holds, and chunk overlap repeats some text between neighbouring chunks so ideas cut at a boundary are not lost.

[Concurrency](https://broadrouter.com/glossary/concurrency/)

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

[Exponential backoff](https://broadrouter.com/glossary/exponential-backoff/)

Exponential backoff is a retry strategy in which a client waits longer after each failed attempt, typically doubling the delay up to a cap, before trying again. Adding random jitter to each wait spreads out retries from many clients so they do not hit the server at the same moment.

[finish\_reason](https://broadrouter.com/glossary/finish-reason/)

finish\_reason is a field on each choice in a chat completion response that says why the model stopped generating. In the OpenAI format the values are stop, length, tool\_calls, content\_filter and the deprecated function\_call. Anthropic's Messages API reports the same idea as stop\_reason.

[Hybrid search](https://broadrouter.com/glossary/hybrid-search/)

Hybrid search is a retrieval method that runs keyword search, such as BM25, and vector search over embeddings on the same query, then merges the two ranked lists into one, often with reciprocal rank fusion (RRF) or a weighted sum of normalized scores.

[Idempotency key](https://broadrouter.com/glossary/idempotency-key/)

An idempotency key is a unique value, usually a random ID such as a UUID, that a client attaches to a request that changes state, such as a payment or a purchase. If the same request is sent again with the same key, the server returns the original result instead of performing the action twice.

[Inference provider](https://broadrouter.com/glossary/inference-provider/)

An inference provider is a company that runs machine learning models, often open-weight LLMs, on GPUs it owns, leases or rents, and sells access to them through an API, usually priced per token, per request or per unit of time. It handles serving, scaling and hardware.

[JSON mode](https://broadrouter.com/glossary/json-mode/)

JSON mode is an LLM API setting, response\_format {"type": "json\_object"} in OpenAI-style APIs, that makes the model's reply valid, parseable JSON. Unlike structured outputs with a JSON Schema, it does not guarantee which keys, types or values the object contains.

[JSON Schema](https://broadrouter.com/glossary/json-schema/)

JSON Schema is a vocabulary, written in JSON, for describing and validating the structure of JSON data: types, required fields, allowed values and nesting. The current version is 2020-12. LLM APIs use it to define tool parameters and the exact shape a structured output must follow.

[Latency](https://broadrouter.com/glossary/latency/)

Latency, for a language model API, is the time a request takes from being sent to the final output token being received. It is the sum of time to first token and generation time, and is usually reported as percentiles such as p50 and p95 rather than as an average.

[LLM router](https://broadrouter.com/glossary/llm-router/)

An LLM router is software or a service that sits between an application and one or more language model APIs and decides, for each request, which model or provider handles it, using rules such as cost, latency, availability or the type of task.

[MCP (Model Context Protocol)](https://broadrouter.com/glossary/mcp/)

MCP (Model Context Protocol) is an open protocol, built on JSON-RPC 2.0, that standardises how LLM applications connect to external tools and data. An MCP server exposes tools, resources and prompts; an MCP client inside the host application discovers them and passes tool calls from the model to the server.

[Model deprecation](https://broadrouter.com/glossary/model-deprecation/)

Model deprecation is when an API provider announces that a model will stop being offered, usually naming a retirement or shutdown date and a recommended replacement. After that date, requests to the model fail, so applications must migrate to another model ID beforehand.

[Model fallback](https://broadrouter.com/glossary/model-fallback/)

Model fallback is a reliability pattern where a failed LLM request is retried on a different model or provider from an ordered list. The trigger can be an error, a rate limit, a context-length error or a content-policy refusal. It runs in your client or in a gateway in front of the APIs.

[Multi-agent system](https://broadrouter.com/glossary/multi-agent-system/)

A multi-agent system is an LLM application in which several agents, each with its own prompt, tools and context, work on parts of one task. A common design is orchestrator-worker: a lead agent plans, starts worker agents in parallel and combines their results.

[OpenAI-compatible API](https://broadrouter.com/glossary/openai-compatible-api/)

An OpenAI-compatible API is an HTTP API that accepts the same request and response formats as the OpenAI API, such as POST /v1/chat/completions, so OpenAI SDKs and tools can use it by changing only the base URL, the API key and the model name.

[Parallel tool calls](https://broadrouter.com/glossary/parallel-tool-calls/)

Parallel tool calls are when a model returns more than one tool call in a single response, for example three file reads at once. In the OpenAI format they arrive as several entries in tool\_calls, and parallel\_tool\_calls set to false limits the model to zero or one call per turn.

[RAG (retrieval-augmented generation)](https://broadrouter.com/glossary/rag/)

RAG (retrieval-augmented generation) is a pattern in which an application first retrieves passages relevant to a question from an external store, such as a search or vector index, and then adds them to the prompt so a language model can answer from that text rather than from memory alone.

[Rate limit](https://broadrouter.com/glossary/rate-limit/)

A rate limit is a cap an API places on how much a client can use it in a time window, such as requests per minute or tokens per minute. Requests over the limit are refused, usually with HTTP 429 Too Many Requests, until usage falls back under it.

[ReAct (reason and act)](https://broadrouter.com/glossary/react-agent/)

ReAct (reason and act) is a prompting pattern, introduced by Yao et al. in 2022, in which a language model interleaves reasoning traces with actions such as search or tool calls. It reads each observation, updates its plan, and repeats until it can give a final answer.

[Request timeout](https://broadrouter.com/glossary/request-timeout/)

A request timeout is the longest time an HTTP client waits for a server before it abandons the request. Clients usually split it into a connect timeout for opening the connection and a read timeout for waiting on data. LLM calls need generous read timeouts because long answers take time.

[Reranking](https://broadrouter.com/glossary/reranking/)

Reranking is a second retrieval stage in which a more accurate but slower model, usually a cross-encoder, scores each query and candidate document pair from a first-pass search and reorders them, so only the most relevant few are passed to the language model.

[Right-to-left (RTL) text](https://broadrouter.com/glossary/rtl-text/)

Right-to-left (RTL) text is writing, such as Arabic or Hebrew, displayed from right to left. Computers store it in logical reading order, and the Unicode Bidirectional Algorithm decides display order when RTL text mixes with left-to-right text, digits, model names or code.

[Seed parameter](https://broadrouter.com/glossary/seed-parameter/)

The seed parameter is an integer you send with a request so the API tries to sample the same tokens when the same request is repeated. OpenAI describes it as best effort: determinism is not guaranteed, and the system\_fingerprint field shows when the backend configuration has changed.

[Server-sent events (SSE)](https://broadrouter.com/glossary/sse/)

Server-sent events (SSE) is a web standard for a server to push a stream of text events to a client over one long-lived HTTP response with the content type text/event-stream. Each event is one or more field lines, such as data:, ended by a blank line.

[Serverless inference](https://broadrouter.com/glossary/serverless-inference/)

Serverless inference is a way to run machine learning models in which the platform starts, scales and stops the compute behind an endpoint automatically, often down to zero when idle, and bills for usage rather than for reserved machines. The trade-off is possible cold-start delay.

[SLA (service level agreement)](https://broadrouter.com/glossary/sla/)

An SLA (service level agreement) is a contractual commitment from a provider to a measurable level of service, usually a monthly uptime percentage, with a defined remedy, usually service credits, if it is missed. An SLA defines what counts as downtime, what is excluded and how to claim.

[Stop sequences](https://broadrouter.com/glossary/stop-sequences/)

Stop sequences are strings you send with a request that make the model stop generating as soon as it produces one of them. OpenAI calls the field stop and accepts up to 4 sequences; Anthropic calls it stop\_sequences. The matched text is normally left out of the returned output.

[Streaming](https://broadrouter.com/glossary/streaming/)

Streaming is a way of returning a language model response in small chunks while it is still being generated, instead of as one reply at the end. OpenAI-compatible APIs stream over server-sent events when the request sets stream to true.

[Structured output](https://broadrouter.com/glossary/structured-output/)

Structured output is an LLM API feature that constrains a model's reply to a format the developer specifies, usually a JSON Schema, so code can parse it reliably. It is stricter than JSON mode, which guarantees valid JSON but not a particular shape.

[Subagent](https://broadrouter.com/glossary/subagent/)

A subagent is a separate agent that a main agent starts to handle one focused task. It runs with its own context window, system prompt and tool access, does the work, and returns only a result or summary to the main agent, which keeps the main conversation small.

[System prompt](https://broadrouter.com/glossary/system-prompt/)

A system prompt is the instruction text an application sends to a language model ahead of the conversation, usually as a message with the system or developer role. It sets the model's role, rules, tone and output format, and the model is meant to follow it over conflicting user messages.

[Throughput](https://broadrouter.com/glossary/throughput/)

Throughput, in LLM serving, is the total amount of work a system completes per unit of time across all concurrent requests, usually measured as output tokens per second or requests per second. It differs from per-stream speed, which is how fast a single request receives its tokens.

[Time to first token (TTFT)](https://broadrouter.com/glossary/time-to-first-token/)

Time to first token (TTFT) is the time from sending a request to a language model until the first output token arrives. It includes network time, queueing on the server and prefill, the step where the model reads the whole prompt, so it grows with prompt length and load.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[tool\_choice](https://broadrouter.com/glossary/tool-choice/)

tool\_choice is a request parameter that controls tool use. In OpenAI's Chat Completions it can be auto (the model decides), required (it must call one or more tools), none (no tools), or a named function to force. Anthropic's equivalent options are auto, any, tool and none.

[Unicode normalization](https://broadrouter.com/glossary/unicode-normalization/)

Unicode normalization converts text to a standard form so that strings that look the same use the same code points. The four forms are NFD, NFC, NFKD and NFKC; the K forms also fold compatibility characters, such as Arabic presentation forms and ligatures, into ordinary letters.

[Vector database](https://broadrouter.com/glossary/vector-database/)

A vector database is a data store that holds embedding vectors, often with their source text and metadata, and answers nearest-neighbour queries: given a query vector, it returns the stored vectors most similar to it, usually by cosine distance, Euclidean distance or inner product.

## Models and tokens

[Arabic diacritics (tashkeel)](https://broadrouter.com/glossary/arabic-diacritics/)

Arabic diacritics, or tashkeel, are small marks written above or below letters to show short vowels, doubled consonants and the absence of a vowel. They are optional: most Arabic text, including news, omits them, so readers and software infer the vowels from context.

[Arabic dialect identification](https://broadrouter.com/glossary/arabic-dialect-identification/)

Arabic dialect identification is the task of deciding which Arabic variety a text or recording is in, such as Egyptian, Gulf, Levantine, Maghrebi or Modern Standard Arabic, at the level of region, country or city. It is used to route messages, pick prompts and split evaluation sets.

[Arabic text normalization](https://broadrouter.com/glossary/arabic-text-normalization/)

Arabic text normalization is the step that maps spelling variants of the same word to one form before matching: removing diacritics and tatweel, unifying alef forms, and often mapping ى to ي and ة to ه. It is applied to search keys and evaluation, not to text you display.

[Arabizi](https://broadrouter.com/glossary/arabizi/)

Arabizi is Arabic written informally in Latin letters, using digits for Arabic sounds with no Latin equivalent, such as 3 for ع. It is common in chat and social media, is mostly dialectal, has no standard spelling, and is often mixed with English or French.

[AWQ (activation-aware weight quantization)](https://broadrouter.com/glossary/awq/)

AWQ (activation-aware weight quantization) is a post-training method that compresses an LLM's weights to low precision, usually 4-bit integers. It uses activation statistics from a small calibration set to find the most important weight channels and scales them so they lose less accuracy. It needs no backpropagation.

[Byte pair encoding (BPE)](https://broadrouter.com/glossary/bpe/)

Byte pair encoding (BPE) is a subword tokenization algorithm that starts from single characters or bytes and repeatedly merges the most frequent adjacent pair into a new vocabulary entry, until it reaches a target vocabulary size. Most current LLM tokenizers use BPE or a byte-level variant.

[Chain-of-thought (CoT)](https://broadrouter.com/glossary/chain-of-thought/)

Chain-of-thought (CoT) is a prompting technique in which a language model is shown or asked to write intermediate reasoning steps before its final answer. It improves accuracy on multi-step problems such as arithmetic and logic, and reasoning models now generate it automatically.

[Chat template](https://broadrouter.com/glossary/chat-template/)

A chat template is the formatting rule, usually written in Jinja and shipped with a model's tokenizer, that converts a list of chat messages with roles into the single token sequence the model expects, including its special control tokens. Using the wrong template can sharply degrade a model's answers.

[Context compaction](https://broadrouter.com/glossary/context-compaction/)

Context compaction is the practice of summarizing a conversation that is nearing the context window limit and continuing the session from that summary, so an agent keeps key decisions and state while dropping raw history such as old tool output.

[Context engineering](https://broadrouter.com/glossary/context-engineering/)

Context engineering is the work of choosing and maintaining the set of tokens a language model sees on each request, including system prompt, tool definitions, history, retrieved documents and notes, so the model gets the smallest set of high-signal information it needs to do the task well.

[Context rot](https://broadrouter.com/glossary/context-rot/)

Context rot is the gradual loss of accuracy and reliability that language models show as their input gets longer, even when the task stays the same and the input is well within the context window. The term was popularised by a 2025 Chroma technical report that tested 18 models.

[Context window](https://broadrouter.com/glossary/context-window/)

A context window is the maximum number of tokens a language model can work with in a single request. It covers the prompt, conversation history, tool definitions, images and the generated output, including any reasoning tokens. Anything that does not fit must be cut, summarised or split across several requests.

[Dialectal Arabic](https://broadrouter.com/glossary/dialectal-arabic/)

Dialectal Arabic is the everyday spoken Arabic of a region or city, such as Egyptian, Gulf, Levantine or Maghrebi Arabic. Dialects differ from Modern Standard Arabic in sounds, words and grammar, have no standard spelling, and are increasingly written online, which makes them harder for language models.

[Distillation (knowledge distillation)](https://broadrouter.com/glossary/distillation/)

Distillation, or knowledge distillation, is a training method in which a smaller student model learns to reproduce the outputs of a larger teacher model, such as its probability distributions or generated answers, so the student keeps much of the teacher's quality at a lower cost to run.

[Embeddings](https://broadrouter.com/glossary/embeddings/)

Embeddings are lists of floating-point numbers (vectors) that a model produces to represent a piece of text, an image or other input. Inputs with similar meaning get vectors that are close together, so distance between vectors can rank search results, cluster documents or find related items.

[Eval (LLM evaluation)](https://broadrouter.com/glossary/eval/)

An eval (short for evaluation) is a repeatable test of a language model or prompt: a fixed set of inputs, a description of what a good output looks like, and a grader (code, a human or another model) that scores each output so that versions can be compared.

[Few-shot prompting](https://broadrouter.com/glossary/few-shot-prompting/)

Few-shot prompting is a technique in which a prompt includes a small number of worked input-output examples before the real input, so a language model infers the task and format from them. The model's weights do not change; zero-shot uses no examples and one-shot uses one.

[Fine-tuning](https://broadrouter.com/glossary/fine-tuning/)

Fine-tuning is the process of continuing to train a pre-trained language model on a smaller, task-specific dataset, such as example conversations, so that its weights change and it performs that task or style more reliably without needing long instructions or examples in every prompt.

[FP8](https://broadrouter.com/glossary/fp8/)

FP8 is an 8-bit floating-point number format used in deep learning, defined in two encodings: E4M3, with 4 exponent and 3 mantissa bits, and E5M2, with 5 exponent and 2 mantissa bits. It halves memory compared with 16-bit formats and runs natively on recent GPUs.

[Frequency and presence penalty](https://broadrouter.com/glossary/frequency-and-presence-penalty/)

Frequency and presence penalties are sampling settings that lower the score of tokens already in the output. The presence penalty applies once a token has appeared at all; the frequency penalty grows with each repeat. In OpenAI's API both range from -2.0 to 2.0, with 0 as the default.

[GGUF](https://broadrouter.com/glossary/gguf/)

GGUF is a binary file format for storing language models for inference with GGML-based runtimes such as llama.cpp. One GGUF file holds the weights, often quantized, plus the metadata needed to load the model, including tokenizer and architecture details, and it can be memory-mapped for fast loading.

[Grounding](https://broadrouter.com/glossary/grounding/)

Grounding is the practice of connecting a language model's output to specific, verifiable sources of information, such as supplied documents, search results or database records, and instructing it to answer from them. Grounded answers can be traced back to a source and are less likely to contain made-up facts.

[Hallucination](https://broadrouter.com/glossary/hallucination/)

A hallucination is output from a language model that sounds plausible but is false, made up or not supported by the input it was given, such as an invented citation, a wrong number or a fabricated API. It comes from the model predicting likely text rather than checking facts.

[Hamza](https://broadrouter.com/glossary/hamza/)

Hamza (ء) is the Arabic letter for the glottal stop. It is written on its own or on a seat: alef (أ إ), waw (ؤ) or yeh (ئ), and آ combines alef with madda. Because typed text is inconsistent about it, search engines and speech scoring often fold the alef forms into ا.

[Image tokens](https://broadrouter.com/glossary/image-tokens/)

Image tokens are the input tokens a vision language model uses to represent an image in a prompt. Providers compute them from the image's size, usually by splitting it into fixed-size patches or tiles after resizing, and bill them like text input tokens, so larger images cost more.

[INT4](https://broadrouter.com/glossary/int4/)

INT4 is a 4-bit integer number format that can represent 16 levels. In LLM inference it usually means 4-bit weight-only quantization (W4A16): weights are stored as INT4 with scale factors and expanded to 16-bit during computation, using about a quarter of the memory of 16-bit weights.

[KV cache](https://broadrouter.com/glossary/kv-cache/)

A KV cache is the memory where a transformer language model stores the attention keys and values computed for every token it has already processed, so each new token reuses them instead of recomputing the whole sequence. It grows with context length and often limits how many requests a GPU can serve.

[LLM benchmark](https://broadrouter.com/glossary/llm-benchmark/)

An LLM benchmark is a fixed, usually public set of tasks with a scoring method, used to measure and compare language models on one skill, such as knowledge questions, coding or speech recognition. A benchmark score shows relative ability on that test, not how a model will do on your own workload.

[LLM-as-a-judge](https://broadrouter.com/glossary/llm-as-judge/)

LLM-as-a-judge is an evaluation method in which a language model grades other model outputs, either scoring a single answer against a rubric or picking the better of two answers. It approximates human judgement at much lower cost but has known biases, such as favouring the first or longer answer.

[Logprobs](https://broadrouter.com/glossary/logprobs/)

Logprobs are the log probabilities a language model assigned to the tokens it generated. An API that returns them shows, for each output token, how likely the model judged it and, optionally, the most likely alternative tokens at that position.

[LoRA (low-rank adaptation)](https://broadrouter.com/glossary/lora/)

LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pre-trained model's weights and trains a pair of small low-rank matrices for selected layers. The learned update can be stored as a small adapter or merged into the weights, adding no inference latency.

[Lost in the middle](https://broadrouter.com/glossary/lost-in-the-middle/)

Lost in the middle is the observed tendency of language models to use relevant information best when it appears near the beginning or end of a long input and worst when it sits in the middle. It was named in a 2023 study by Liu et al.

[max\_tokens](https://broadrouter.com/glossary/max-tokens/)

max\_tokens is an LLM API parameter that sets the most tokens a model may generate in one response. When generation reaches the limit, it stops and the response is marked as cut off, for example with finish\_reason set to "length".

[Mixture of experts (MoE)](https://broadrouter.com/glossary/mixture-of-experts/)

Mixture of experts (MoE) is a neural network design in which a layer holds many expert sub-networks and a learned router sends each token to only a few of them. An MoE language model has a large total parameter count but uses a small active subset for each token.

[Model license](https://broadrouter.com/glossary/model-license/)

A model license is the legal terms under which a model's weights, code and outputs may be used, changed and shared. Open-weight models use standard software licences such as Apache 2.0 or MIT, or custom model licences that can add conditions, such as limits on large commercial services.

[Modern Standard Arabic (MSA)](https://broadrouter.com/glossary/msa/)

Modern Standard Arabic (MSA) is the standardized form of Arabic used in writing, news, education and formal speech across the Arab world. It is nobody's native everyday language: people speak regional dialects such as Egyptian or Gulf Arabic and switch to MSA in formal settings.

[Multimodal AI](https://broadrouter.com/glossary/multimodal/)

Multimodal AI is a model or system that works with more than one type of data, such as text, images, video and audio. A multimodal model might read a photo and a question together, or take speech in and answer in text. A vision-language model is the most common kind.

[Named entity recognition (NER)](https://broadrouter.com/glossary/named-entity-recognition/)

Named entity recognition (NER) is the task of finding names in text and labelling each with a type, such as person, organization or location. Classic systems tag every token; LLMs are often asked instead to return a list of entities as JSON, which is then checked against the source.

[Open-weight model](https://broadrouter.com/glossary/open-weight-model/)

An open-weight model is a machine learning model whose trained weights are published for anyone to download and run, usually under a licence with conditions. It differs from fully open-source AI, which also releases the training code and detailed information about the training data, so others can study and reproduce the model.

[Optical character recognition (OCR)](https://broadrouter.com/glossary/ocr/)

Optical character recognition (OCR) is technology that detects and converts printed or handwritten text in an image, scan or PDF into machine-readable characters. Classic OCR outputs text and its position on the page; vision-language models can read text and answer questions about it directly.

[Prefill and decode](https://broadrouter.com/glossary/prefill-and-decode/)

Prefill and decode are the two phases of LLM inference. In prefill the model processes the whole prompt in parallel and fills its KV cache; in decode it generates the reply one token at a time, reusing that cache. Prefill mostly sets time to first token; decode sets output speed.

[Prompt engineering](https://broadrouter.com/glossary/prompt-engineering/)

Prompt engineering is the process of writing, structuring and testing the instructions given to a language model so that it consistently produces output that meets your requirements. It covers role messages, clear instructions, examples, formatting, and measuring results against test cases.

[Quantization](https://broadrouter.com/glossary/quantization/)

Quantization is the process of storing a neural network's numbers, mainly its weights, in a lower-precision format such as 8-bit or 4-bit instead of 16-bit, so the model uses less memory and often runs faster. Good methods keep the accuracy loss small.

[Reasoning tokens](https://broadrouter.com/glossary/reasoning-tokens/)

Reasoning tokens are tokens a reasoning model generates to work through a problem before it writes its visible answer. They are usually billed as output tokens, count toward the output limit and the context window, and may be hidden or summarized by the API.

[Speculative decoding](https://broadrouter.com/glossary/speculative-decoding/)

Speculative decoding is an inference technique in which a small, fast draft model proposes several next tokens and the large target model verifies them in a single forward pass, keeping the accepted ones. It speeds up generation without changing the distribution of the output.

[Taa marbuta](https://broadrouter.com/glossary/taa-marbuta/)

Taa marbuta (ة, U+0629) is an Arabic letter that appears only at the end of a word, often marking feminine nouns and adjectives. It looks like heh (ه, U+0647) with two dots, so typed text mixes the two, and many search normalizers map ة to ه.

[Temperature](https://broadrouter.com/glossary/temperature/)

Temperature is a sampling parameter that controls how random a language model's output is. The model divides its token scores by the temperature before turning them into probabilities: low values make the likeliest token dominate, high values flatten the distribution so less likely tokens are chosen more often.

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Tokenizer](https://broadrouter.com/glossary/tokenizer/)

A tokenizer is the component that converts text into the sequence of integer token IDs a language model reads, and converts generated IDs back into text. Each model ships its own tokenizer, so the same text can produce a different number of tokens on different models.

[Tokens per second](https://broadrouter.com/glossary/tokens-per-second/)

Tokens per second (TPS) is the rate at which a language model generates output tokens, usually measured for one request after its first token arrives. It sets how fast a reply streams once it starts. Summed across all concurrent requests it becomes system throughput; the wait before the first token is measured separately.

[Top-p (nucleus sampling)](https://broadrouter.com/glossary/top-p/)

Top-p, also called nucleus sampling, is a decoding method in which a language model samples the next token only from the smallest set of most likely tokens whose combined probability reaches a threshold p, such as 0.9, and discards the unlikely tail.

[Vision-language model](https://broadrouter.com/glossary/vision-language-model/)

A vision-language model (VLM) is a model that takes images and text together as input and produces text, so it can describe pictures, answer questions about them and read documents, charts or screenshots. VLMs are also called image-text-to-text or multimodal models.

[vLLM](https://broadrouter.com/glossary/vllm/)

vLLM is an open-source library for LLM inference and serving, first developed at UC Berkeley's Sky Computing Lab. It manages KV cache memory with PagedAttention, batches incoming requests continuously, supports many quantization formats and exposes an OpenAI-compatible HTTP server.

[Vocabulary size](https://broadrouter.com/glossary/vocabulary-size/)

Vocabulary size is the number of distinct tokens in a language model's tokenizer, each with its own ID and embedding. A larger vocabulary can encode common text in fewer tokens, but it makes the embedding and output layers bigger, and it only helps for text the vocabulary was trained on.

## Pricing and billing

[Cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/)

Cached input tokens are prompt tokens that an LLM provider has recently processed and can reuse from a cache instead of computing again, usually a repeated prefix such as a system prompt or document. Providers bill them at a lower rate than new input tokens.

[Flat-rate LLM plan](https://broadrouter.com/glossary/flat-rate-llm-plan/)

A flat-rate LLM plan charges a fixed price for a period, such as a month, for access to a language model API instead of billing per token. Such plans usually limit something else, such as concurrency, speed, models or usage windows, so the caps matter as much as the price.

[GPU-hour](https://broadrouter.com/glossary/gpu-hour/)

A GPU-hour is one graphics processor used for one hour. Cloud providers bill rented GPU instances by the hour or second, and researchers use GPU-hours to state how much compute a job took. You pay for the machine's time, whether or not it is busy.

[Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/)

Input tokens are the tokens a language model reads in a request, including the system prompt, conversation history, tool definitions and images. Output tokens are the tokens it generates in reply, including any reasoning. LLM APIs usually price them separately, and output usually costs more.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Prepaid credit](https://broadrouter.com/glossary/prepaid-credit/)

Prepaid credit is money added to an API account before use. Each request deducts its cost from the balance, and when the balance runs out, metered requests stop until you top up. It caps spend at what you have paid in.

[Spend limit](https://broadrouter.com/glossary/spend-limit/)

A spend limit is a cap on how much an API account, project or key may spend in a period, usually a month. When usage reaches it, the provider rejects further requests, often with an HTTP 429 or 400 error, until the period resets or the limit is raised.

## Privacy and security

[API key](https://broadrouter.com/glossary/api-key/)

An API key is a secret string that identifies and authorizes a client calling an API. LLM APIs usually expect it in an Authorization: Bearer header, and anyone who holds the key can make requests that are billed to the account that owns it.

[Data processing agreement (DPA)](https://broadrouter.com/glossary/dpa/)

A data processing agreement (DPA) is the contract between a controller and a processor that governs how the processor handles personal data. GDPR Article 28(3) requires it and lists its contents: documented instructions, confidentiality, security, subprocessor rules, deletion or return of data, and audits.

[Data residency](https://broadrouter.com/glossary/data-residency/)

Data residency is a requirement or commitment that data is stored and processed in a specific country or region. For an LLM API it covers where prompts are processed during inference and where any stored content sits, and it often excludes account and billing data.

[GDPR](https://broadrouter.com/glossary/gdpr/)

The GDPR (General Data Protection Regulation, Regulation (EU) 2016/679) is the EU law on processing personal data. It has applied since 25 May 2018 and covers EU-established organisations and some outside the EU. For LLM APIs it governs processor contracts, international transfers and data minimisation.

[Guardrails](https://broadrouter.com/glossary/guardrails/)

Guardrails are checks that an application runs around a language model call to keep it within set limits. Input guardrails screen what goes into the model, output guardrails check what comes out, and tool guardrails validate actions before they run. A failed check blocks, rewrites or flags the request.

[IP allow list](https://broadrouter.com/glossary/ip-allow-list/)

An IP allow list is a security control that accepts requests only from listed IP addresses or CIDR ranges and refuses all others. On an API, it limits the damage from a leaked key, because the key does not work from networks that are not on the list.

[Jailbreak (LLM)](https://broadrouter.com/glossary/jailbreak/)

A jailbreak is an input crafted to make a language model ignore its safety training or the rules its deploying application set, so that it produces content or takes actions it would normally refuse. OWASP treats jailbreaking as a form of prompt injection.

[PDPL (Saudi and UAE data protection laws)](https://broadrouter.com/glossary/pdpl/)

PDPL (Personal Data Protection Law) is the common name for Saudi Arabia's personal data law, issued by Royal Decree M/19 and enforced by SDAIA, and for the UAE's Federal Decree-Law No. 45 of 2021. Both regulate processing of personal data and set conditions for transfers abroad, including prompts sent to foreign LLM APIs.

[PII (personally identifiable information)](https://broadrouter.com/glossary/pii/)

PII (personally identifiable information) is information that can be used to distinguish or trace a person's identity, such as a name, ID number or biometric record, plus other information linked or linkable to that person. The GDPR's broader term is personal data. In LLM work, PII redaction removes it from prompts before they leave your systems.

[Prompt injection](https://broadrouter.com/glossary/prompt-injection/)

Prompt injection is an attack in which text supplied to a language model, by a user or hidden in a web page, email, file or tool result, changes the model's behaviour against the developer's intent. OWASP ranks it first (LLM01:2025) in its Top 10 for LLM Applications.

[Subprocessor](https://broadrouter.com/glossary/subprocessor/)

A subprocessor is a third party that a processor engages to carry out processing of personal data on behalf of the controller. For an LLM API, subprocessors can include GPU hosts, partner inference providers and backup capacity. Under GDPR Article 28 they need the controller's authorisation and the same data protection obligations.

[Zero data retention](https://broadrouter.com/glossary/zero-data-retention/)

Zero data retention (ZDR) is a data-handling policy in which an AI provider processes prompts and outputs only to serve a request and does not store that content afterward, in logs, caches or training data. Content-free records such as token counts and timings are usually still kept.

## Speech

[Character error rate (CER)](https://broadrouter.com/glossary/character-error-rate/)

Character error rate (CER) measures transcription or OCR accuracy at the character level: the substitutions, deletions and insertions needed to turn the output into the reference text, divided by the number of characters in the reference. Lower is better, and 0 is a perfect match.

[Code-switching](https://broadrouter.com/glossary/code-switching/)

Code-switching is the use of two or more languages, or varieties of one language, within the same conversation or sentence, in speech or writing. Examples include Arabic speakers mixing English words into dialect, or moving between Modern Standard Arabic and their dialect.

[Ogg Opus](https://broadrouter.com/glossary/ogg-opus/)

Ogg Opus is audio encoded with the Opus codec (RFC 6716) and stored in an Ogg container (RFC 7845). It is built for speech and interactive audio at low bitrates, and it is the format WhatsApp uses for voice notes, usually as .ogg or .opus files.

[Sample rate](https://broadrouter.com/glossary/sample-rate/)

Sample rate is the number of audio samples captured per second, measured in hertz (Hz). Speech recognition models commonly expect 16,000 Hz (16 kHz) mono audio. Together with bit depth and channel count, the sample rate sets how large an uncompressed audio file is per minute.

[Speaker diarization](https://broadrouter.com/glossary/diarization/)

Speaker diarization is the task of dividing an audio recording into segments by speaker, answering "who spoke when". It labels each stretch of speech with an anonymous speaker ID such as SPEAKER\_00, and is usually combined with a transcript to show which person said each line.

[Speech-to-text](https://broadrouter.com/glossary/speech-to-text/)

Speech-to-text, also called automatic speech recognition (ASR), is technology that converts spoken audio into written text. A speech-to-text API accepts an audio file or stream and returns a transcript, usually priced per minute of audio. Accuracy is measured with word error rate and varies with accent, dialect and recording quality.

[SRT subtitles](https://broadrouter.com/glossary/srt-subtitles/)

SRT (SubRip) is a plain-text subtitle format. Each caption is a sequence number, a start and end time written as hh:mm:ss,mmm --> hh:mm:ss,mmm, one or more lines of text, and a blank line. WebVTT is the similar web format used by HTML video.

[Text-to-speech (TTS)](https://broadrouter.com/glossary/text-to-speech/)

Text-to-speech (TTS), or speech synthesis, converts written text into spoken audio. A TTS API takes text, a voice and an output format such as MP3 or WAV, and returns generated speech. It is the reverse of speech-to-text, which turns audio into text.

[Timestamps (speech-to-text)](https://broadrouter.com/glossary/timestamps-asr/)

Timestamps in speech-to-text are start and end times, in seconds, attached to parts of a transcript. Segment timestamps cover a phrase or sentence; word timestamps cover each word. They are needed for subtitles, search inside recordings and aligning text with speakers.

[Voice activity detection (VAD)](https://broadrouter.com/glossary/vad/)

Voice activity detection (VAD) is a signal-processing step that classifies short frames of audio as speech or non-speech. Speech systems use it to skip silence, split long recordings at pauses, detect when a speaker has finished, and avoid transcribing background noise.

[WAV](https://broadrouter.com/glossary/wav/)

WAV (Waveform Audio File Format, or WAVE) is an audio container built on Microsoft and IBM's RIFF format. It usually holds uncompressed linear PCM samples, so its size is predictable: sample rate times bytes per sample times channels, per second.

[Word error rate](https://broadrouter.com/glossary/word-error-rate/)

Word error rate (WER) is the standard accuracy measure for speech recognition: the number of word substitutions, deletions and insertions needed to turn a transcript into the correct reference, divided by the number of words in the reference. Lower is better.

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference API reference](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary&utm_content=sources)
