LLM inference, end to end

You type a question. Why is my API returning a 503? You press enter. And about a second later, the answer starts typing itself, word by word. Feels like one single step, right? It's not. It's actually a relay race. Your question crosses around ten different systems, two very different kinds of computers, and three different caches, before that first word comes back to you. So in this video, we are going to slow all of that down. We'll follow your one question, hop by hop, from your screen to the GPU, and all the way back. And by the end, you'll know three things. Where the time actually goes. Why the answer streams word by word. And where a 503 is actually born.

But first, what is inference, really? Every AI model has two lives. The first life is training. That's when the model studies a huge amount of text, for weeks, often on thousands of GPUs, and slowly sets its numbers. The second life is inference. That's when the finished model is actually used, to answer one request at a time. Think of it like this. Training is studying for years. Inference is answering the exam question that's in front of you right now. And during inference, the model learns nothing from your question. Its numbers are frozen. It only computes an answer. Now here's the surprising part. A language model really does only one thing. It looks at all the text so far, and it guesses the next small piece of text. That piece is called a token. It adds that token, and then it guesses again, and again. A full answer is just hundreds of these tiny guesses, in a row. Training happens once per model. But inference happens every single time anyone, anywhere, sends a message. So inference is where the speed, the cost, and the outages really live. And the software that runs these guesses fast, for thousands of people at once, is called an inference engine. That's exactly what we're opening up today. Our example model for the day is an open model called Llama three, eight B. Eight billion numbers. Keep that in mind, we'll come back to it.

Okay, so before we start, what is a model, physically? It's a file. A very big file of numbers, called weights. For our model, that's about eight billion numbers, roughly sixteen gigabytes. And running the model just means multiplying your input by those numbers, billions of times. That's why we need GPUs. A GPU has thousands of small cores and very fast memory, built for exactly this kind of math. Now look at the whole trip from ten thousand feet. Almost everything here runs on ordinary CPUs. Web servers, gateways, databases. Only one stop actually runs the model, and that one needs a GPU. GPUs are expensive, so they do just one job. Here are the ten stops. Your app. The API gateway. The prompt builder. The input guardrails. The LLM gateway. The inference router. Then the inference engine, on the GPU. And on the way back, the output guardrails, the stream back to you, and finally, the work that happens after the answer. Watch the caches too. There are three of them, in three different places. A response cache at the gateway, that can skip the GPU completely. A prefix cache inside the engine, that skips re-reading a prompt it has already seen. And the KV cache, also inside the engine, that saves the model from re-reading your whole conversation for every new word. And there are two numbers we'll keep coming back to. Time to first token, or TTFT. That's how long until the first word shows up. And time per output token, or TPOT. That's the gap between every word after that. Think of a restaurant. TTFT is how long the first dish takes. TPOT is how fast the rest keep coming.

Stop one, your app. When you press enter, your app doesn't just send your sentence. It sends a small package of JSON. A list of messages, each with a role, like system, user and assistant. The name of the model. And a setting called stream, set to true. Most chat APIs today use this same shape. Why stream? Imagine the full answer takes ten seconds to write. Without streaming, you stare at a spinner for ten seconds. With streaming, the first word shows up in about half a second, and you start reading while the model is still writing. The total time is the same. It just feels so much faster. Under the hood, the connection stays open, and the server pushes small chunks to you, one after another. These are called server sent events. Your app also adds a few things you never see. Your login token, a conversation ID, and a request ID. That request ID will follow us all the way down, so later we can trace every single hop. And before any AI happens at all, there's the normal internet trip. Look up the address, set up a secure connection, maybe pass through an edge server close to you, and then reach the data center. That's usually tens of milliseconds, just in travel.

Stop two, the API gateway. This is the front door. It unlocks the encryption, checks your API key, and figures out which customer you are. Then it checks your limits. And here's an important detail. LLM gateways don't just count requests, they count tokens. Because one request can be a hundred tokens, and another can be a hundred thousand. Go over your tokens per minute, and you get a 429. Too many requests. Remember that one, because it's not the 503 we're hunting. The gateway also checks that the request makes sense. Is the body too big? Are these settings even allowed? Bad requests get turned away right here, before they cost anything.

Stop three, the prompt builder. And here's something most people don't realise. Your question is the smallest part of what the model actually reads. The app stacks up a system prompt with instructions, your past conversation, maybe some documents, the tools the model is allowed to use, and then, at the very end, your question. Because the model has no memory between calls. None. If it seems to remember what you said earlier, that's only because the app sends the whole history again, every single time. And when that history gets too long, the app trims it, or summarises the old parts. Now, if the app uses RAG, retrieval augmented generation, a few more things happen here. First, it may rewrite your question to make it easier to search. Then it embeds it. A small model turns your text into a list of numbers that captures its meaning. And yes, that's an inference call too. Then a vector database finds the chunks of text with the closest meaning. A reranker reads them more carefully and picks the best few. And those chunks get pasted into the prompt. Think of a librarian who reads your question, runs to the shelves, and hands the model the three most useful pages. And if tools are involved, like an agent that can check your pod status, the model might answer with a tool call instead of text. The app runs the tool, adds the result to the prompt, and calls the model again. So one question from you can mean several full trips through everything we're about to see.

Stop four, input guardrails. Before we spend any GPU time, we check what's going in. Is there personal data that should be masked? Is someone trying a prompt injection, like hiding instructions inside a document? Is the topic even allowed? And these checks should also run on the retrieved documents, not just on what you typed. Why here? Because saying no now costs a few milliseconds of CPU. Saying no after the model has written a whole answer costs real GPU seconds. The cheapest place to say no is before the GPU.

Stop five, the LLM gateway. Think of it as a smart switchboard for models. An easy question can go to a small, cheap model. A hard one goes to a big model. And if a model, or a whole provider, is failing, it can retry, or fall back to another one. This is also where our first cache lives, the response cache. If someone asked this exact question recently, the gateway can just return the saved answer. No GPU at all. Some gateways even match questions with a similar meaning. That's faster, but be careful, a similar question doesn't always deserve the same answer. And it keeps the books. Every token going in, and every token coming out, is counted per team. Because that's exactly what you get billed for.

Stop six, the inference router. The same model is usually running on many GPU servers at once. We call each one a replica. So which one should get your request? Plain round robin works badly here, because one request might be ten tokens long, and the next one might be fifty thousand. So modern routers look inside. Does this replica already have your system prompt in its cache? Send it there, and it skips work. How long is its queue? How full is its memory? Projects like the Gateway API Inference Extension and llm-d do exactly this on Kubernetes. And if a company runs many fine-tuned versions of one model, called LoRA adapters, the router also has to pick a server that has the right adapter. Now, replicas don't appear instantly. When traffic grows, a new replica has to download the model weights, about sixteen gigabytes for our model, load them into GPU memory, warm up, and pass a readiness check. That can take minutes. And now, finally, here's where a 503 is born. Maybe there's no ready replica, because the new ones are still starting up. Maybe every replica's queue is already full, so the system starts rejecting new work. Or maybe the GPUs have simply run out of memory for more conversations. So when your API says 503, it usually doesn't mean the model is broken. It means there's no free seat on any GPU right now.

Stop seven. We're on the GPU server now, inside the inference engine, something like vLLM or SGLang. First job, turn all those messages into one long piece of text, using the model's own chat template. Special marker tokens show where each message starts, who is speaking, and where it ends. Then the tokenizer chops that text into tokens. Here's the real output of the Llama three tokenizer, on our question. Why. Space is. Space my. Space API. Space returning. Then a lonely space. Then 503, as its own piece. And the question mark. Eight tokens. And each token is really just a number, an ID, in a vocabulary of about 128,000 pieces. Common words are a single token. Rare words get split. The model never sees letters. Only these numbers. One more check. The prompt's tokens, plus the maximum number of tokens it's allowed to write, have to fit inside the model's context length. If they don't, the request is rejected, or the oldest part gets cut.

Next, the scheduler. And honestly, this is the most important part that nobody explains. Your request doesn't go straight to the GPU. It joins a waiting queue, together with requests from lots of other users. Older servers used static batching. Grab a group of requests, run them together, and wait until every single one is finished before starting the next group. Like a bus that only leaves when every passenger has reached their stop. Modern engines use continuous batching instead. The batch is rebuilt at every single step. Finished requests hop off, new ones hop on, and the GPU never sits idle. This idea came from a system called Orca, back in twenty twenty two. There's another trick here, called chunked prefill. If a new request arrives with a huge prompt, processing it all at once would freeze everyone who's in the middle of an answer. So the engine cuts the long prompt into slices, and mixes them in, step by step, while everyone else keeps getting their words. And before any request runs, the scheduler books memory for it, for something called the KV cache, which we'll meet properly in a minute. If memory runs out, it has to pause someone, and redo their work later. Keep that in mind. Memory is the real limit on how many people one GPU can serve.

Okay. Now let's zoom all the way in, to one foot, inside the model itself. First, each token ID looks up its own row in a giant table, and becomes a list of 4,096 numbers. That list is called a vector, and it's how the model holds the meaning of that token. Then that vector goes through a layer. Step one, RMS norm. It simply rescales the numbers, so they stay in a healthy range. Step two, attention. This is the famous part. Every token makes three new vectors. A query, which asks, what am I looking for? A key, which says, here's what I contain. And a value, the actual information it will hand over. Each token compares its query with the keys of every earlier token, never the later ones, because the model can't peek at the future. Those scores become percentages, and the token takes that mix of values. Think of a library. Your query is the question you walk in with. The keys are the labels on the spines. And the values are the pages inside the books. Two clever details here. The position of each word is mixed into the queries and keys, by rotating them. That's called RoPE, and it's why dog bites man and man bites dog mean different things. And Llama uses grouped query attention. Thirty two query heads share just eight sets of keys and values. That makes the memory we'll need four times smaller. After attention, the original vector is added back on top. That's called a residual connection, so nothing important gets lost. Then step three, another norm, and a feed forward network, where each token thinks on its own. Llama uses one called SwiGLU. And in mixture of experts models, a router sends each token to just a few experts, out of many. And that's one layer. Our model stacks thirty two of them, one after another. At the very end, the last position is turned into logits. One score for every single token in the vocabulary. About 128,000 scores, one for each possible next piece. Bigger models don't fit on one GPU. A seventy billion parameter model is spread across several GPUs, using tensor parallelism. Each GPU computes a slice of every layer, and after each layer, they quickly share their results before moving on.

Now we can finally see what happens to your request. It runs in two phases. Phase one is prefill. All the tokens of your prompt go through all thirty two layers at the same time, in parallel. That's a lot of math at once, which is exactly what GPUs love. And at the end of prefill, out comes the very first token of the answer. That moment is your time to first token. During prefill, every layer also saves the key and the value of every token. That's the KV cache. Think of it as taking notes while you read, so you never have to re-read the whole book just to write the next word. And it's not small. Let's do the math. Two, for keys and values, times thirty two layers, times eight heads, times one hundred and twenty eight numbers, times two bytes each. That's one hundred and twenty eight kilobytes, for every single token. So one long conversation, of about eight thousand tokens, needs around one gigabyte of GPU memory, just for its notes. Phase two is decode. Now the model writes one token at a time. Only the newest token goes through the thirty two layers. It reads all the notes in the KV cache, adds its own, and produces the next token. Then again, and again. And here's why decode is the slow part. Each step does very little math, but the GPU still has to read all sixteen gigabytes of weights from memory. On an H100, that alone takes roughly five milliseconds, every single step. So the limit isn't math anymore, it's memory speed. That's your time per output token. It's like a chef who walks to the pantry and back, for every single word. The fix is batching. One pantry trip can cook for many tables at once. And how do you store all those notes without wasting memory? vLLM's answer was PagedAttention. Instead of reserving one big block for each conversation, the KV cache is split into small pages of sixteen tokens, like hotel rooms that are booked only when needed. A little table keeps track of which rooms belong to whom. So almost no memory is wasted, and many more users fit. Prefix caching builds on that. If your request starts with the same system prompt as an earlier one, those pages are simply reused, and prefill gets much shorter. That's exactly the saving behind the prompt caching discounts that API providers offer. Now put it all together. An eighty gigabyte GPU holds the sixteen gigabytes of weights. Most of what's left becomes KV cache. At about one gigabyte per long conversation, that's roughly fifty long chats at the same time. That's the real ceiling. And when it's full, requests wait, or get rejected. 503, again. And this also explains pricing. Your input is read in one parallel prefill pass. But every output token needs its own full trip through the model. That's why most price lists charge more for output tokens than for input tokens.

Back to those hundred and twenty eight thousand scores. How does the model pick the next word? First, the scores can be adjusted. Penalties for repeating itself, and if you asked for JSON, a mask that blocks every token that would break valid JSON. Then softmax turns the scores into probabilities that add up to one hundred percent. Temperature decides how adventurous to be. Top k and top p cut away the unlikely options. And then one token is picked at random, by those probabilities. At temperature zero, it simply takes the top one. When does it stop? When the model produces its special end token, when it writes one of your stop strings, or when it hits your max tokens limit. And if the model wrote a tool call, a parser turns that text into a proper, structured tool call for your app. Reasoning models add one more twist. Before the visible answer, they first generate thinking tokens, using this exact same decode loop. You wait for those tokens, and on most APIs, you pay for them as output tokens too. And why can the same question give different answers? With temperature above zero, the randomness is on purpose. But even at temperature zero, answers can change slightly, because how many other requests share your batch changes the exact math on the GPU. Researchers have shown this, and how to fix it, at some cost in speed.

So here's every big speed trick, and what it actually fixes. Continuous batching keeps the GPU busy. PagedAttention stops memory waste, so more users fit. Prefix caching cuts the time to first token on repeated prompts. Chunked prefill stops long prompts from freezing everyone else. FlashAttention computes attention without moving huge tables in and out of slow memory. CUDA graphs cut the overhead of launching thousands of tiny GPU tasks. Speculative decoding lets a small draft model guess a few tokens ahead, and the big model checks them all in one step. Quantization shrinks the weights and the cache, so there's less to read. And disaggregated serving runs prefill and decode on separate GPUs, because they need such different things.

Now let's ride back up. Every new token ID goes to the detokenizer immediately, not at the end. It turns the ID back into text. But here's a fun detail. Some characters, like an emoji, are split across several tokens. The rocket emoji is three separate pieces in Llama's tokenizer. So the detokenizer holds those bytes back until the character is complete. It also holds back any text that might be the start of a stop string. Stop eight, output guardrails. They check the answer while it streams, in small windows, instead of waiting for the whole thing. Stop nine, the stream back to you. Each chunk travels as a server sent event. The finish reason comes last. And usage, if you ask for it, comes in one extra chunk. One warning for my DevOps friends. A proxy like nginx buffers responses by default, and that can hold back every chunk until the very end, which kills streaming. And long answers can hit idle timeouts, which show up as a 504. And if you close the tab halfway through? A well built stack notices, cancels the request, and frees those KV cache pages right away, for the next person.

Stop ten, after the answer. The conversation is saved, so it can be sent again next time. Your tokens are billed. And the whole journey shows up as one trace, with one ID across all ten stops. On the dashboard, watch time to first token, time per output token, queue length, and KV cache usage. Queue length is usually the first thing to climb, before the 503s arrive. So here's the error map. A four hundred means your request didn't fit, for example it's too long for the context window. A 429 means you went over your rate limit. A 503 means there's no free capacity right now. And a 504 usually means something timed out on a long stream. Let's watch the whole thing one more time, at full speed. Your app. The gateway. The prompt builder. The guardrails. The LLM gateway. The router. Tokenize, schedule, prefill, and the first token. Then decode, token by token, streaming back to you. That's inference. One question, ten stops, and hundreds of tiny guesses. Next time an answer streams in, you'll know exactly what just happened. This was supraj.dev. See you in the next one.