What Actually Happens When You Call an LLM API

I spend my workdays keeping distributed systems from lying to each other. Reading how LLM serving actually works felt like coming home. Start with the illusion. The model is stateless. It remembers nothing between messages. Every time you send a turn, the application rebuilds the entire conversation and sends it again. Your browser holds the history locally, the backend persists it in a database, and the full transcript is reconstructed per request. The memory you experience lives in your app, not in the model. Once you see that, the rest of the stack is a series of familiar moves. ## The path The request leaves your browser and hits an application backend, which owns the conversation store. From there it goes to an inference API, which rebuilds the prompt and hands it to a router. The router's job is sticky: send this request to the same node that served your last one, because that node already has your prefix cached. The node then does the work in two phases, a prefill pass over your tokens and a decode pass that generates the reply one token at a time. ## Three levers, all older than LLMs **The KV cache** means the model never recomputes what it already saw. That is memoization. **Continuous batching** keeps the GPU fed. In a static batch the whole group waits for its slowest sequence, finished requests cannot return, and new arrivals queue until the batch drains. Orca (OSDI 2022) fixed that with iteration-level scheduling: the scheduler runs one iteration of the model at a time, so a finished request can leave and a new one can join at a token boundary. That is preemptive scheduling at a finer granularity, applied to a workload nobody had scheduled that way before. It is often called continuous batching, and it is the single change that moved throughput most. **Sticky routing** sends you back to a warm cache. That is session affinity. None of these are new ideas. They are the same playbook, run under much more load and a much higher bill. ## The one genuinely stateful box on the inference path Your history is state, but your application owns it. On the inference path itself the KV cache is the only thing that remembers, and it remembers for minutes, only so that the next request is cheaper. Everything else, your history, your preferences, the feeling that the agent knows you, is a database your application maintains and resends. My day job is estimating what algorithms cost on hardware, and the interesting question in LLM serving is rarely the model. It is the scaling curve. Caches, routing affinity, queue management, batch scheduling. Impressive engineering, not emerging consciousness. ## What that means if you are building on these APIs The model is the stateless part and you own the state. The quality of memory your users feel is the quality of your retrieval and re-assembly, not a property you bought from the provider. That re-assembly step, what you resend and what you drop, is where most agent products are actually won or lost. The magic is that there is no magic. There is a cache, a queue, and a bill.