Engineering articles
Updated
Production engineering for LLM APIs: streaming, 429 retries, JSON output, tool calling, prompt caching, long context, latency and logging every request. 33 articles, each opening with a direct answer. Every outside fact links to its source with the date we checked it.
A private endpoint starts with a clear workload
Before you ask for a private endpoint, write down the workload: requests per day and at peak, tokens in and out, the concurrency that implies, the context you need, the processing region, the model and its licence, and a budget. Agree region and limits in writing, then smoke-test the issued endpoint before real data.
Before fine-tuning: define the problem to solve
Fine-tune only after an evaluation shows a recurring failure that better prompts, examples, retrieval and validation did not fix. Missing knowledge usually needs retrieval, and format errors need structured output and checks. Before asking for a custom model, collect labelled examples you may legally use, hold out a test set, and record the score to beat.
BroadNet's GPU hardware and serving approach, as BroadNet describes it
BroadNet says it runs one serving stack on three hardware tiers: data-center GPUs from A100 to B300, a workstation tier of RTX and DGX Spark cards that carries the always-on public API, and appliances on client sites. It reports about 270 models run across 13 GPU families, NVIDIA and AMD. Some API models are served by US inference partners instead.
Cache LLM API responses to cut cost: an exact-match cache in Python
Hash the full request (model, messages, temperature, max_tokens, response_format, tools, reasoning settings and your prompt version) into a key, and store the answer under it with an expiry. Serve a stored answer only for deterministic, side-effect-free requests. Measure how often identical requests repeat in your logs first: that share is the most an exact cache can save.
How to fix invalid JSON from an LLM: causes, repair, retry
Check finish_reason first: a reply cut off at max_tokens cannot be repaired safely, so raise the limit and resend. Otherwise strip code fences and surrounding prose, decode the first JSON value, then fix only unambiguous mistakes such as trailing commas. Validate against a schema, and if that fails, send the exact error back once or twice. Structured output prevents most of this where supported.
LLM API failover: fall back to a second provider in your own code
Keep an ordered list of OpenAI-compatible providers, each with its own client and model ID. Fail over on connection errors, timeouts, 5xx and rate-limit 429s once the SDK's own retries are spent. Never fail over on 400, 401, 402, 403, 413 or 422. Add a circuit breaker, and check that the backup matches the primary on features and data terms.
LLM API rate limits by provider: RPM, TPM and concurrency compared
LLM API rate limits differ in kind, not just size. OpenAI, Anthropic and Gemini cap requests and tokens per minute by usage tier. DeepSeek and DeepInfra cap concurrent requests. Groq and Cerebras free tiers add daily caps. On first paid tiers, OpenAI and Anthropic allow thousands of requests a minute; BroadNet pay-as-you-go keys allow 30.
LLM batch processing without a Batch API: a resumable runner
Without a Batch API, run the job yourself: read items from a JSONL file, cap requests in flight at the key's limit, pace request starts under the per-minute limit, retry 429 and 5xx with capped backoff, and append each result with its ID. On restart, skip finished IDs. The rate limit, not the model, usually sets the finish time.
Make human review part of your document AI workflow
Treat every extracted document as unapproved until rules pass. Send it to a person when a required field is missing, sums disagree, the date or vendor looks wrong, or the model is unsure. Also review a small random share of clean documents, log every correction in your own system, and track the review rate and audited error rate.
Migrating off the OpenAI API: a code walkthrough in Python, Node, cURL
Change the client's base URL, API key and model ID. That is enough for Chat Completions calls. Code on the Responses API must be rewritten to Chat Completions: input becomes messages, output_text becomes choices[0].message.content, and tools lose a level of nesting. Keep embeddings on a separate client if the new provider has none, then run a smoke test before switching traffic.
OpenAI-compatible API base URLs: 14 providers, checked
An OpenAI-compatible API lets you keep the OpenAI SDK and change two things: base_url and the API key. The base URL differs by provider, for example https://api.groq.com/openai/v1 for Groq, https://openrouter.ai/api/v1 for OpenRouter and https://inference.broadnet.ai/v1 for BroadNet Inference. Paths, model IDs and supported parameters still differ, so test one request before you switch.
RAG chunk size and overlap: what they do to retrieval and cost
Start with chunks of about 256 to 512 tokens, 0 to 15% overlap, and top-k tuned on your own queries. Bigger chunks raise the tokens every query sends to the LLM. Overlap multiplies the tokens you embed and store by size ÷ (size − overlap), so 50% overlap doubles the index. Published tests found small or no-overlap chunks often recall as well or better.
Record audio in the browser and send it to a speech-to-text API
Record with getUserMedia and MediaRecorder, picking a format with isTypeSupported, and drop empty chunks. If the API accepts only WAV or MP3, decode the recording in the browser and encode 16 kHz mono WAV, or upload it and convert with ffmpeg on your server. Either way, send it through your own backend so the API key never reaches the browser.
Reduce prompt tokens: measured cuts for system prompts, tools, data
Measure first: log prompt_tokens per call, find the biggest part, and diff with the model's tokenizer. Across five tokenizers, trimming a wordy system prompt saved 68 to 71% and CSV used 31 to 41% fewer tokens than minified JSON. On three models, a trimmed prompt with 3 short tools instead of 10 verbose ones cut the full request by about 65 to 71%.
Setting max_tokens for reasoning models: a budgeting method
On a reasoning model, max_tokens caps thinking and answer together, so a tight limit can end the reply before any visible text. Measure total completion tokens per task type, set max_tokens to about the 95th percentile plus 20%, keep prompt plus max_tokens inside the context window, and retry cut-off replies with a doubled budget up to a hard cap.
Text, speech or vision? Start with the task
Start from your input and output. Text in, text out is chat. A photo or scan in, text out is chat with an image, on a model that accepts images. Recorded speech in, text out is transcription; speech to an answer is transcription then chat. BroadNet does not offer text-to-speech, image generation or embeddings.
Transcribe long audio over 25 MB: split, send and stitch
Most transcription APIs reject one file over about 25 MB. Convert the recording to 16 kHz mono first: that alone fits about 13 minutes of WAV or 54 minutes of 64 kbit/s MP3 into BroadNet's 26,214,400-byte limit. For longer audio, split at silences with ffmpeg, transcribe the chunks in parallel and join the text in order.
Uno in vLLM: what BroadNet's serving study measured
Uno in vLLM is a September 2026 preprint by BroadNet (DOI 10.5281/zenodo.22652610). It adds Uno, an adapter that lets a model draft its own tokens, to vLLM. On an H100 with Qwen3-8B, BroadNet measured 2.58 times plain-vLLM throughput at one stream, 2.20 times at eight and 1.74 times at 32, within 3% of the reference implementation.
Uno on Gemma 4 26B: BroadNet's mixture-of-experts study
BroadNet's September 2026 preprint (DOI 10.5281/zenodo.22820511) trains the first Uno adapter for a mixture-of-experts model, Gemma 4 26B A4B, served at 4-bit. The adapter decodes 1.166 times faster than plain decoding on a 72-prompt production workload, behind two separate drafters at 1.474 and 1.553, but leads the production drafter at about 10,500 tokens of context.
Extract invoice, receipt and form data to JSON with a vision LLM
Render each page to an image, send it to a vision model with a prompt that names every field and its format, and ask for JSON only. Then validate in code: parse it, check its schema, check that line items add up to the total, and retry once with the error. Route repeat failures to a person.
How to count tokens before you send an LLM request
Count tokens with the tokenizer of the model you call when it is public; otherwise calibrate a characters-per-token ratio from the usage field of real requests, per language and content type. Add a 10 to 20 percent margin, then check that prompt tokens plus max_tokens fit the context window, leaving room for reasoning tokens on thinking models.
How to evaluate an LLM before switching providers
Collect 50 to 200 real prompts with known good outcomes. Score both models with code checks first and an LLM judge only where code cannot decide. Swap answer order to cancel position bias, use a judge from a third model family, check it against your own labels, and fix the pass rule, including cost and latency, before you run it.
How to handle LLM API 429 errors: backoff, jitter and limiters
A 429 means you hit a rate, concurrency or spend limit. Retry it with capped exponential backoff and random jitter, and honour Retry-After when sent. Then prevent most 429s: cap requests in flight with a semaphore at the key's limit and queue the rest. Retry 5xx the same way; fix other 4xx errors instead.
How to structure prompts so the prefix caches
Order every request from most stable to least stable: system prompt, tool definitions, reference documents, conversation history, then the new message. Make each layer byte-identical between calls: no timestamps or IDs at the top, tools sorted, JSON serialised deterministically, and history only ever appended. Then measure cached_tokens to confirm the prefix is actually reused.
JSON mode vs json_schema: structured output that holds up
JSON mode (response_format json_object) only promises valid JSON. Structured output (response_format json_schema) also promises your fields and types, where the provider enforces it. Either way, check finish_reason, parse, and validate against a schema such as a Pydantic model. If validation fails, send the error back and ask again, with a hard limit on attempts.
LLM API key security: storage, rotation and IP allow-lists
Keep LLM API keys in a secret manager, never in code or the browser. Give each app or agent its own key, rotate on a schedule, and lock keys to your servers with an IP allow-list. BroadNet Inference says its allow-lists take 32 entries, revocation stops new requests within five seconds, and keys last one year.
LLM API usage logging: cost per user and feature, PII redacted
Wrap every LLM call in one function that records model, feature, a hashed user ID, prompt, cached, reasoning and output tokens, cost, latency, status and the provider request ID. Compute cost from a price table, not the invoice. Log metadata by default; if you must keep prompt text, redact it before writing and set a short retention.
LLM latency: TTFT vs tokens per second vs end-to-end time
Time to first token (TTFT) is the wait before the first output token. Tokens per second is how fast the answer then streams. End-to-end latency is the total: TTFT plus generation time, roughly output tokens divided by per-user speed. Prompt length and queueing drive TTFT, model size and load drive speed, and answer length drives the total. Measure all three at p50 and p95.
LLM streaming explained: SSE chunks, deltas, usage and cancel
An LLM API streams a chat completion as Server-Sent Events: each data line holds a JSON chunk whose choices[0].delta carries the next piece of text. Concatenate the deltas, stop at data: [DONE], and set stream_options include_usage to get token counts in a final chunk with an empty choices list. To cancel, close the connection.
Long context vs RAG: when 1M tokens replaces a vector DB
Long context replaces RAG when the corpus fits in the window, changes rarely and questions need many parts of it at once; caching keeps repeat queries cheap. RAG still wins when the corpus outgrows the window, updates constantly, needs per-user permissions or must answer in a second or two. Many systems retrieve with RAG and reason with long context.
Switching to an OpenAI-compatible API: a migration checklist
To move to an OpenAI-compatible API, change three settings (base_url, api_key and model) and then check what differs: model IDs, how reasoning tokens count toward max_tokens, how streaming reports usage, which errors to retry, rate limits and key security. For BroadNet Inference the base URL is https://inference.broadnet.ai/v1.
Tool calling with open models: schemas, parallel calls, retries
Open models on OpenAI-compatible APIs use the same tools and tool_calls format as OpenAI. Keep schemas small and flat, parse arguments as JSON strings, validate them before running anything, and return errors to the model as tool results so it can retry. Cap the steps. On BroadNet, GLM 5.3 and Qwen3.8 27B list tool support.
When a 1M-token context window is worth it (and when not)
A 1M-token context window lets one request hold a whole codebase, a contract set or hours of transcripts, so the model sees everything at once instead of retrieved fragments. It pays off for cross-file reasoning and one-off analysis; for repeated lookups over a large, changing corpus, retrieval is cheaper. DeepSeek V4.1 Flash and GLM 5.3 accept 1,048,576 tokens.
















