TOPIC AI
Multimodal Models AI models that understand text, images, audio, and video together
IN 10 SECONDS
Multimodal models (GPT-4V, Claude 3, Gemini) can process and reason across multiple data types simultaneously. You can show them a screenshot and ask 'What's the error here?', give them a diagram and ask them to generate Terraform code, or show a video and ask for a summary. The model encodes all modalities into a shared representation space.
GOTCHA Multimodal models can misread images — they may interpret handwriting wrong, miss small text, or hallucinate details not in the image. Always verify model outputs against the actual image. For screenshots with small text, pre-process with OCR before sending to the model.
HOW A MULTIMODAL MODEL PROCESSES AN IMAGE
01 Input the user sends: 'What's wrong with this Dockerfile?' + an image of the Dockerfile.
02 Vision encoder the image is split into patches and processed through a vision transformer (ViT) into embeddings.
03 Projection layer image embeddings are projected into the same latent space as text embeddings via a connector module.
04 Language model the combined text + image embeddings are processed through the transformer — the model 'sees' the image and generates text about it.
POKE IT YOURSELF
curl -X POST -d '{"model":"gpt-4-vision-preview","messages":[{"role":"user","content":[{"type":"text","text":"What's in this image?"},{"type":"image_url","image_url":{"url":"https://example.com/screenshot.png"}}]}]}' https://api.openai.com/v1/chat/completions — send an image to GPT-4 Vision
ollama run llava 'Describe this image' --image photo.jpg — run multimodal model locally with Ollama
Drill this topic →
~80 sec read