How a batched forward pass writes the KV cache

gpu-tips, note 1. November 19th, 2026.

An inference engine such as vLLM runs many requests in one forward pass. The requests have different lengths, and each request has its own KV cache. This note shows how the new tokens of all requests go into one batch, and where the keys (K) and values (V) of each token go.

Each layer writes the K and V of a new token directly into the cache blocks of that token's request. No step cuts the KV out of an output tensor after the pass.

The example

Three requests are in the engine at the same time.

RequestTokens in the cacheNew tokens in this stepReason
A5,0001A generates its next token.
B8001B generates its next token.
C02,000C is a new prompt. All its tokens are new.

This step has 2,002 new tokens.

The flat batch

The engine lays out the new tokens as one flat batch, [A, B, C1, C2, …, C2000]. The input to each layer has 2,002 rows, one row for each new token.

The cache is one large pool of blocks on the GPU, and all requests share it. In vLLM, one block holds the K and V of 16 tokens by default. A slot is the place of one token in a block. The slot number is the block number × 16 + the position in the block.

The engine passes two index tables with the batch. The names are the names that vLLM uses.

TableContentUsed to
Slot mappingFor each row, the cache slot where its K and V goWrite
Block tableFor each request, the list of cache blocks that it ownsRead

One layer, step by step

The figure shows the batch, the pool and the two tables. The pool in the figure has 600 blocks. Use the buttons to go through one layer. Point at a row or a block to see where its data is.

Flat batch, 2,002 rows
KV block pool on the GPU, 600 blocks of 16 slots
A B C free A block with an outline is not full.
Block table
One block

The two outputs

A forward pass has two outputs. The engine cuts only one of them.

Why the attention is not one square matrix

A plain batched attention needs all requests at the same length, which is 5,001 here. B then carries 4,200 empty positions and C carries 3,001. Approximately half of the 15,003 positions are empty.

All requests padded to 5,001 positions
A
5,001 in use
B
801 in use, 4,200 empty
C
2,000 in use, 3,001 empty

15,003 positions. 7,201 of them (48%) are empty.

With the block table
A
5,001 in 313 blocks
B
801 in 51 blocks
C
2,000 in 125 blocks

7,802 positions. None is empty.

empty position (padding)

The block table removes the empty positions. Each row reads only the blocks that exist for its request. Thus requests of very different lengths share one batch, and no position is empty.

What follows from the block pool

Prefix hits on the GPU

Let request D run at the same time as C. The first 1,024 tokens of the two prompts are the same. These tokens fill 64 blocks, and the 64 blocks are in the pool already. The engine puts those blocks into the block table of C and removes those 1,024 rows from the batch. It copies nothing and computes nothing for these tokens.

A hit always has a whole number of blocks. The engine identifies each full block by a hash of its tokens and of all the tokens before it.

64 blocks, 1,024 tokens, reference count 2
Blocks of D
D is not in the engine.
Blocks of C
block that C computes in this step block in the pool already, shared other block of D
Rows of C in the batch
Rows in the batch
K and V copied for the hit
0

Blocks shared by reference

The 64 blocks are now in two block tables. Each block has a reference count, which is 2 here. The engine releases a block only when no request uses it. A released block keeps its data until the engine needs the space. Thus a later request with the same prefix can still get a hit.

LMCache

LMCache enters at the same place. It copies a loaded chunk into pool blocks, and those blocks go into the block table of the request. A layer reads these blocks in the same way as all other blocks.

Where the prefix isWhat the engine doesRows in the batch
In blocks of the GPU poolPuts the blocks into the block table. It copies nothing.Removed
In a chunk that LMCache holdsCopies the chunk into pool blocks. Then puts the blocks into the block table.Removed
In no cacheComputes K and V in each layer and writes them to the slots.Kept

Back to gpu-tips or Home

My past life... here