BroadRouter
Get started

Multimodal AI

Updated

Multimodal AI is a model or system that works with more than one type of data, such as text, images, video and audio. A multimodal model might read a photo and a question together, or take speech in and answer in text. A vision-language model is the most common kind.

Also called: multimodal model, multimodal AI model, multimodal input, multi-modal, modalities

“Multimodal” tells you what a model can take in. Always check the exact list, because two models both called multimodal can accept quite different inputs.

How it works

Each input type is turned into the same thing the language model already reads: a sequence of vectors in its context window. Text goes through the tokenizer. An image goes through a vision encoder and becomes a block of image tokens. Video is usually handled as a series of frames, and audio through an audio encoder.

Model cards show how specific this gets, even within one family:

Model Inputs listed on its model card Output
Gemma 4 26B A4B text, image, video as frames; no audio (Google lists audio only for the E2B, E4B and 12B sizes) text
Qwen3.8 27B text, image, video text

APIs also split the work by endpoint. OpenAI’s audio guide, for example, sends file transcription to a speech-to-text endpoint and points chat apps that need audio to a separate audio-in-chat feature.

The narrower term for text plus images is vision-language model.

Why it matters

  • Cost. Non-text inputs are converted to tokens or billed per unit of media. On BroadNet a larger image uses more input tokens, so resize to the detail you need.
  • Architecture. If no single model accepts every input you have, chain models: speech-to-text first, then a text or vision model.
  • Quality checks. Reading a printed invoice, a chart or a handwritten note are different skills. Test the modality on your own documents, not only on a demo image.

On BroadNet Inference

BroadNet’s docs list these inputs per model:

Model Input
DeepSeek V4.1 Flash text and images
GLM 5.3 text
Qwen3.8 27B text and vision
BroadNet Gemma 4 VL 26B text and images
BroadNet Transcribe Arabic / BroadNet Transcribe audio, on /v1/audio/transcriptions

Images go to /v1/chat/completions as a data: URL in the message content list; the Arabic invoice guide has a full example. Audio is a separate request: WAV or MP3 up to 26,214,400 bytes, with language set, as in the WhatsApp voice notes guide. A voice-note assistant therefore makes two calls: transcribe, then chat.

Questions

Is a vision-language model multimodal?

Yes. A vision-language model is the most common kind of multimodal model, taking text and images. Multimodal is the wider term and also covers audio and video.

Can I send audio to a chat model on BroadNet?

BroadNet's docs list audio only on the transcription endpoint. Transcribe the audio first, then send the text to a chat model.

Do images cost more than text?

On BroadNet pay-as-you-go keys an image is billed as input tokens at the model's input price, and a larger image uses more tokens. On a plan, images are part of the flat price.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity