TOPIC AI
Model Serving putting a model behind an endpoint
IN 10 SECONDS
Model serving means deploying a trained model behind an HTTP/gRPC endpoint (like vLLM, Triton) that dynamically batches queries to optimize graphics card (GPU) utility.
GOTCHA Deploying standard Python libraries (Flask/FastAPI) to wrap model generation code blocks GPU calls, limiting concurrency to one query at a time.
HOW DYNAMIC BATCHING WORKS
01 Client queries multiple users send text prompts to the serving endpoint simultaneously.
02 Batch engine vLLM groups prompts into a single matrix block (continuous batching).
03 GPU execution evaluates batched matrix queries in a single hardware cycle.
04 Stream response streams tokens back to users asynchronously as they complete.
POKE IT YOURSELF
python -m vllm.entrypoints.openai.api_server --model facebook/opt-125m — start an OpenAI-compatible serving host locally with vLLM
Drill this topic →
~70 sec read