gpu-tips, note 1. November 19th, 2026.
An inference engine such as vLLM runs many requests in one forward pass. The requests have different lengths, and each request has its own KV cache. This note shows how the new tokens of all requests go into one batch, and where the keys (K) and values (V) of each token go.
Each layer writes the K and V of a new token directly into the cache blocks of that token's request. No step cuts the KV out of an output tensor after the pass.
Three requests are in the engine at the same time.
| Request | Tokens in the cache | New tokens in this step | Reason |
|---|---|---|---|
| A | 5,000 | 1 | A generates its next token. |
| B | 800 | 1 | B generates its next token. |
| C | 0 | 2,000 | C is a new prompt. All its tokens are new. |
This step has 2,002 new tokens.
The engine lays out the new tokens as one flat batch, [A, B, C1, C2, …, C2000]. The input to each layer has 2,002 rows, one row for each new token.
The cache is one large pool of blocks on the GPU, and all requests share it. In vLLM, one block holds the K and V of 16 tokens by default. A slot is the place of one token in a block. The slot number is the block number × 16 + the position in the block.
The engine passes two index tables with the batch. The names are the names that vLLM uses.
| Table | Content | Used to |
|---|---|---|
| Slot mapping | For each row, the cache slot where its K and V go | Write |
| Block table | For each request, the list of cache blocks that it owns | Read |
The figure shows the batch, the pool and the two tables. The pool in the figure has 600 blocks. Use the buttons to go through one layer. Point at a row or a block to see where its data is.
A forward pass has two outputs. The engine cuts only one of them.
A plain batched attention needs all requests at the same length, which is 5,001 here. B then carries 4,200 empty positions and C carries 3,001. Approximately half of the 15,003 positions are empty.
15,003 positions. 7,201 of them (48%) are empty.
7,802 positions. None is empty.
The block table removes the empty positions. Each row reads only the blocks that exist for its request. Thus requests of very different lengths share one batch, and no position is empty.
Let request D run at the same time as C. The first 1,024 tokens of the two prompts are the same. These tokens fill 64 blocks, and the 64 blocks are in the pool already. The engine puts those blocks into the block table of C and removes those 1,024 rows from the batch. It copies nothing and computes nothing for these tokens.
A hit always has a whole number of blocks. The engine identifies each full block by a hash of its tokens and of all the tokens before it.
The 64 blocks are now in two block tables. Each block has a reference count, which is 2 here. The engine releases a block only when no request uses it. A released block keeps its data until the engine needs the space. Thus a later request with the same prefix can still get a hit.
LMCache enters at the same place. It copies a loaded chunk into pool blocks, and those blocks go into the block table of the request. A layer reads these blocks in the same way as all other blocks.
| Where the prefix is | What the engine does | Rows in the batch |
|---|---|---|
| In blocks of the GPU pool | Puts the blocks into the block table. It copies nothing. | Removed |
| In a chunk that LMCache holds | Copies the chunk into pool blocks. Then puts the blocks into the block table. | Removed |
| In no cache | Computes K and V in each layer and writes them to the slots. | Kept |
My past life... here