TOPIC AI
Model Routing sending each prompt to the best model for the job
IN 10 SECONDS
Model routing is the practice of dynamically selecting which LLM handles a given request based on criteria like complexity, cost, latency, or domain. A lightweight classifier (or simple heuristics) analyzes the incoming prompt and routes it — 'What time is it?' goes to a cheap, fast model; 'Debug this distributed system deadlock' goes to the most capable one. This optimizes spend and response quality simultaneously.
GOTCHA A misrouted prompt is worse than no routing at all — sending a complex reasoning task to a cheap model produces garbage that looks confident. Start with conservative routing (send ambiguous cases to the strongest model) and tighten thresholds as you collect data.
HOW A MODEL ROUTER DECIDES
01 User prompt arrives at the routing layer: 'Explain the CAP theorem with examples.'
02 Classifier evaluates complexity, token count, and domain tags — rates this as 'medium complexity, technical.'
03 Routing rules match the classification to a model tier: simple → GPT-4o-mini, medium → Claude 3.5 Sonnet, hard → GPT-4o or Opus.
04 Selected model receives the prompt and generates the response. The router logs which model was used, latency, and cost for optimization.
POKE IT YOURSELF
python -c "from litellm import Router; router = Router(model_list=[...]); router.completion(model='gpt-4o', messages=[...])" — use LiteLLM's router for multi-model routing
Drill this topic →
~75 sec read