“Multimodal” tells you what a model can take in. Always check the exact list, because two models both called multimodal can accept quite different inputs.
How it works
Each input type is turned into the same thing the language model already reads: a sequence of vectors in its context window. Text goes through the tokenizer. An image goes through a vision encoder and becomes a block of image tokens. Video is usually handled as a series of frames, and audio through an audio encoder.
Model cards show how specific this gets, even within one family:
| Model | Inputs listed on its model card | Output |
|---|---|---|
| Gemma 4 26B A4B | text, image, video as frames; no audio (Google lists audio only for the E2B, E4B and 12B sizes) | text |
| Qwen3.8 27B | text, image, video | text |
APIs also split the work by endpoint. OpenAI’s audio guide, for example, sends file transcription to a speech-to-text endpoint and points chat apps that need audio to a separate audio-in-chat feature.
The narrower term for text plus images is vision-language model.
Why it matters
- Cost. Non-text inputs are converted to tokens or billed per unit of media. On BroadNet a larger image uses more input tokens, so resize to the detail you need.
- Architecture. If no single model accepts every input you have, chain models: speech-to-text first, then a text or vision model.
- Quality checks. Reading a printed invoice, a chart or a handwritten note are different skills. Test the modality on your own documents, not only on a demo image.
On BroadNet Inference
BroadNet’s docs list these inputs per model:
| Model | Input |
|---|---|
| DeepSeek V4.1 Flash | text and images |
| GLM 5.3 | text |
| Qwen3.8 27B | text and vision |
| BroadNet Gemma 4 VL 26B | text and images |
| BroadNet Transcribe Arabic / BroadNet Transcribe | audio, on /v1/audio/transcriptions |
Images go to /v1/chat/completions as a data: URL in the message content list; the Arabic invoice guide has a full example. Audio is a separate request: WAV or MP3 up to 26,214,400 bytes, with language set, as in the WhatsApp voice notes guide. A voice-note assistant therefore makes two calls: transcribe, then chat.