Two speeds of a token: prefill, decode, and the latency you actually ship

We build production GenAI for operations, knowledge, and data. That sentence is a latency budget, not a slogan. A mapping queue, a retrieval desk, and an agent that has to finish do not meet the model as one speed. They meet two. Prefill reads the prompt and writes the cache. Decode emits the tokens someone actually waits through. Fold both into one response-time number and the next incident sends you shopping for the wrong fix.

A token does not cost the same on the way in as on the way out. Prefill is usually compute-bound, and it sets time to first token. Decode is usually memory-bandwidth-bound, and it sets inter-token latency. A tiny prompt can still be bandwidth-bound, with too little parallel work to hide the weight load. A wide decode batch can climb back toward compute, because more sequences finally give the math units something to do.

Prefill is a burst. Decode is a walk.

Prefill runs the prompt as a wide parallel pass. Attention and the feed-forward blocks do real arithmetic across those tokens together, so the GPU's math units stay busy. A long system pack, a fat tool schema, or a retrieved page shows up as silence. Time to first token, TTFT, is queue time plus this burst, plus the sample of the first token.

Decode is a thin step, repeated. The model loads weights and reads the key-value cache for everything already said. Arithmetic per step is small next to the bytes moved, so the machine waits on memory. Inter-token latency, ITL, is the gap between steps. Blank, then a lump: TTFT. Starts, then stutters: ITL.

Stretch the prompt and the burst grows. Stretch the answer and you multiply the ticks. Which phase dominates is a property of the request, not of the model card.

Read it as a trace. Admission, batching, and any gateway hop sit inside TTFT before a multiply happens. The mark is the handoff. Miss it, and a scheduler problem looks like a bandwidth problem.

The cache is why decode stays expensive

The model stores keys and values for prior tokens instead of recomputing the prompt. That KV cache grows with sequence length, batch width, and depth. Long context is resident memory. A request that fits at a couple thousand tokens can push its neighbors out in the tens of thousands, and a batch that used to fit stops fitting. Grouped-query attention shrinks the cache by sharing key and value heads. Longer still costs more.

Prefix caching is the product form of the same fact. Keep system blocks and tool schemas byte-stable and that prefix's prefill can be reused. The next turn encodes the new evidence. We already score that stability as a harness property. Layout rules live in the cache-hit note. A cold prefix inflates TTFT on every agent turn. A hot prefix aims the burst at what changed.

A pinned prefix still occupies memory. The compute you saved is capacity you spent. Prefill writes that cache and every decode step reads it, so a blended price per token hides the line that caps concurrency.

Batch the lane. Chunk the long prompt.

Static batching fills a rectangle, runs it lockstep, and holds finished lanes until the longest ends. Continuous batching, sometimes called in-flight batching, admits a new request the moment a slot frees. Decode slots are the scarce resource. Filling them turns bandwidth into throughput.

A long prompt can still seize the machine and hitch every decode already running. The paragraph stalls. The model did not pause to think. The scheduler let a compute burst preempt the bandwidth lane. Chunked prefill splits that prompt and interleaves the pieces with decode steps already in flight. TTFT for the long prompt stretches. ITL for the other lanes stays closer to the promise. Under load, that bargain usually beats a frozen batch. A separate prefill pool only scales the two speeds apart.

A wider decode batch raises arithmetic intensity, which helps throughput, until each request's ITL climbs because the step is shared. Tokens per second can rise while the cursor stutters. Those are different SLOs.

The bars are an illustrative footprint: more tokens kept hot, fewer lanes that fit. Trust a dashboard over the drawing only once it shows the same split.

What a single number hides

One histogram. End-to-end percentiles point at the GPU when the queue missed, or at the batcher when the prefix was cold.

Context treated as free. Weights fit, so the limit climbs. KV does not. The batch shrinks and both speeds get worse.

Chunking left off. Continuous batching is on, and one retrieval prompt still hitches every stream. The lane was interrupted.

A quant with no gate. ITL looks healthier. Tool-call JSON starts failing. The serving change was the defect.

Speculation on open text. The draft disagrees. You pay for the proposal and the reject, and the report still looks busy.

Quantize the bytes, then keep the gate

Quantization cuts bytes on the path decode waits on. Lower-precision weights lighten each step. A quantized KV cache shrinks the footprint, so a wider batch or a longer sequence fits before you spill. The loss shows up where a contract is picky: tool arguments that must be valid JSON, a citation that must name a real passage, a code that must land in the allowed set.

Illustrative, not a measured result. Cut the bytes on the decode path and watch the gate. If rejects climb, the failure moved from the memory bus to the review queue. Keep long prefixes, short answers, tool calls, and refusals in the regression set. Compare TTFT, ITL, and gate pass rate on the same traffic shape.

Speculation changes the slope

A cheap draft proposes several next tokens. The target checks them in one step, closer to a short prefill than to a lonely decode. Accepts land as several tokens per pass, and ITL falls. A reject drops the tail and still costs the pass. Templated tool calls agree often. Open-ended reasoning agrees less.

It does not warm a cold prefix. A reject also changes the slot mid-step. If the trace cannot say what was accepted, leave the feature off. The harness still owes a gather, an act, and a gate before done. Measure acceptance on your own traffic first, and keep the fallback at one token at a time.

Two SLOs and a split bill

TTFT is the wait until the first token, as the caller feels it: queue, route, cache lookup, and prefill. Record queue wait beside it, or the remedy will be more GPUs for a batcher problem.

ITL is the gap after the first token. Time per output token averages those gaps. A stutter is an ITL miss even when the first token was prompt. Put both SLOs on a high percentile. An average hides the prompt that froze the lane.

The job adds every tool round trip. Gather, act, and verify can be several model calls. A slow gate is not a slow decode.

The bill splits the same way. Input tokens that paid a full prefill. Cached prefix tokens that did not. Output tokens that paid decode and cache reads. Then goodput: tokens that survived the gate.

Illustrative budgets, not published SLOs. A watched drafting lane might want the first token inside a few hundred milliseconds, and an ITL tight enough that a sentence does not stall. A nightly mapping lane can wait longer on the first token if the window holds and the gate still scores the row. Different products need different numbers, or they steal each other's capacity.

What you ship is the split

Put both speeds on the dashboard. Keep prefixes stable, and count the memory they still hold. Batch continuously, chunk long prefills, quantize behind a gate, and speculate only where your own traffic accepts the draft. Split the bill across input, cache hits, and output.

A faster first token does not attach evidence. A smoother stream does not make a tool legal. At sunrisegenai.com the work is production GenAI that has to finish. The latency you ship is the one a person can feel and a bill can explain. If the dashboard cannot show both, you are still guessing which speed you bought.