---
title: "What does multimodal mean in AI? Text, image and audio"
description: "A multimodal model takes more than one kind of input: text, images, video or audio. What each costs over an API, and how to chain models when one cannot do it all."
url: https://broadrouter.com/glossary/multimodal/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Multimodal AI

Updated 2026-11-16

Multimodal AI is a model or system that works with more than one type of data, such as text, images, video and audio. A multimodal model might read a photo and a question together, or take speech in and answer in text. A vision-language model is the most common kind.

Also called: multimodal model, multimodal AI model, multimodal input, multi-modal, modalities

“Multimodal” tells you what a model can take in. Always check the exact list, because two models both called multimodal can accept quite different inputs.

## How it works

Each input type is turned into the same thing the language model already reads: a sequence of vectors in its [context window](https://broadrouter.com/glossary/context-window/). Text goes through the [tokenizer](https://broadrouter.com/glossary/tokenizer/). An image goes through a vision encoder and becomes a block of image tokens. Video is usually handled as a series of frames, and audio through an audio encoder.

Model cards show how specific this gets, even within one family:

| Model | Inputs listed on its model card | Output |
| --- | --- | --- |
| Gemma 4 26B A4B | text, image, video as frames; no audio (Google lists audio only for the E2B, E4B and 12B sizes) | text |
| Qwen3.8 27B | text, image, video | text |

APIs also split the work by endpoint. OpenAI’s audio guide, for example, sends file transcription to a speech-to-text endpoint and points chat apps that need audio to a separate audio-in-chat feature.

The narrower term for text plus images is [vision-language model](https://broadrouter.com/glossary/vision-language-model/).

## Why it matters

-   **Cost.** Non-text inputs are converted to tokens or billed per unit of media. On BroadNet a larger image uses more input tokens, so resize to the detail you need.
-   **Architecture.** If no single model accepts every input you have, chain models: speech-to-text first, then a text or vision model.
-   **Quality checks.** Reading a printed invoice, a chart or a handwritten note are different skills. Test the modality on your own documents, not only on a demo image.

## On BroadNet Inference

BroadNet’s docs list these inputs per model:

| Model | Input |
| --- | --- |
| DeepSeek V4.1 Flash | text and images |
| GLM 5.3 | text |
| [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) | text and vision |
| [BroadNet Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/) | text and images |
| BroadNet Transcribe Arabic / BroadNet Transcribe | audio, on `/v1/audio/transcriptions` |

Images go to `/v1/chat/completions` as a `data:` URL in the message content list; the [Arabic invoice guide](https://broadrouter.com/guides/read-arabic-invoices-with-vision/) has a full example. Audio is a separate request: WAV or MP3 up to 26,214,400 bytes, with `language` set, as in the [WhatsApp voice notes guide](https://broadrouter.com/guides/transcribe-whatsapp-voice-notes/). A voice-note assistant therefore makes two calls: transcribe, then chat.

## Questions

### Is a vision-language model multimodal?

Yes. A vision-language model is the most common kind of multimodal model, taking text and images. Multimodal is the wider term and also covers audio and video.

### Can I send audio to a chat model on BroadNet?

BroadNet's docs list audio only on the transcription endpoint. Transcribe the audio first, then send the text to a chat model.

### Do images cost more than text?

On BroadNet pay-as-you-go keys an image is billed as input tokens at the model's input price, and a larger image uses more tokens. On a plan, images are part of the flat price.

## See also

-   [Vision-language model](https://broadrouter.com/glossary/vision-language-model/) (term)
-   [Optical character recognition (OCR)](https://broadrouter.com/glossary/ocr/) (term)
-   [Speech-to-text](https://broadrouter.com/glossary/speech-to-text/) (term)
-   [Token](https://broadrouter.com/glossary/token/) (term)
-   [Context window](https://broadrouter.com/glossary/context-window/) (term)
-   [Read Arabic invoices and receipts with a vision model](https://broadrouter.com/guides/read-arabic-invoices-with-vision/) (guide)
-   [Transcribe WhatsApp voice notes to text with an API](https://broadrouter.com/guides/transcribe-whatsapp-voice-notes/) (guide)
-   [Extract invoice, receipt and form data to JSON with a vision LLM](https://broadrouter.com/blog/extract-data-from-documents-with-vision-llm/) (article)
-   [Cheapest vision LLM API: cost per image and per page, worked out](https://broadrouter.com/blog/cheapest-vision-llm-api/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-multimodal&utm_content=page-cta) ·

**Sources**

-   [Hugging Face model card, google/gemma-4-26B-A4B-it (modalities by model size) (checked 2026-11-16)](https://huggingface.co/google/gemma-4-26B-A4B-it)
-   [Hugging Face model card, Qwen/Qwen3.8-27B (image and video input) (checked 2026-11-16)](https://huggingface.co/Qwen/Qwen3.8-27B)
-   [OpenAI, audio and speech guide (transcription vs audio in chat) (checked 2026-11-16)](https://developers.openai.com/api/docs/guides/audio)
-   [BroadNet Inference API reference, models and modalities (checked 2026-11-16)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-multimodal&utm_content=sources)
