gpu-tips, note 2. November 20th, 2026.
The Mooncake paper (FAST '25) published three request traces from a production LLM service. Many KV cache and routing benchmarks replay them. This note shows what one record of the tool-agent trace contains, what the whole trace looks like, and how to turn a record into a request.
The trace has no text. Each record gives the lengths of a request and a list of block hashes. The files are in the Mooncake repository.
The file is JSON Lines. One line is one request. This is the first line of toolagent_trace.jsonl.
{"timestamp": 0, "input_length": 6758, "output_length": 500,
"hash_ids": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]}
| Field | Meaning |
|---|---|
timestamp | Arrival time in milliseconds from the start of the trace. |
input_length | Prompt length in tokens. |
output_length | Number of generated tokens. |
hash_ids | One ID for each 512-token block of the prompt, in order. The number of IDs is the input length divided by 512, rounded up. |
The example has 6,758 input tokens. That is 13 full blocks and one partial block, so the list has 14 IDs.
An ID identifies a block together with all the blocks before it. Two requests that start with the same IDs have the same tokens in that part of the prompt. Thus the KV cache of one request can serve the shared part of the other.
The figure shows the first, second and fourth records of the trace. All three start with block 0. After that block, each request has its own IDs.
The shared block is 512 tokens. It is probably a system prompt that many agents use. A later request of the same agent session repeats a longer list, for example blocks 0 to 13 and then new blocks.
The trace has no session ID. A session is visible only as a sequence of requests whose block lists extend each other.
I computed these values from the file.
| Property | Value |
|---|---|
| Requests | 23,608 |
| Duration | 59 minutes |
| Mean rate | 6.7 requests per second |
| Blocks per request | Median 13, maximum 247 |
| Distinct block IDs | 183,300 of 409,616 block references |
| Blocks that repeat the prefix of an earlier request | 55% |
| Tokens | p10 | p50 | Mean | p90 | p99 | Max |
|---|---|---|---|---|---|---|
| Input | 1,814 | 6,346 | 8,596 | 16,804 | 61,525 | 126,195 |
| Output | 3 | 30 | 182 | 507 | 897 | 2,000 |
The trace gives no text, so the benchmark must make the tokens. This procedure keeps the lengths and the prefix sharing exact.
hash_ids. Cut the result to input_length tokens.prompt of /v1/completions. vLLM accepts token IDs there, so no tokenizer changes the length.max_tokens to output_length.timestamp.Keep the random token IDs below the special tokens of the model. Then no random token ends the turn or starts a tool call.
If a request is longer than the context of the model, keep the leading whole blocks that fit. Do not drop the request. A dropped request also removes the cache entries that later requests of the session use.
No. The prompt is random token IDs. If you decode it, you get unrelated word pieces from many languages.
This is not a problem for cache measurements. The KV cache, the prefix cache of vLLM and LMCache all use the token IDs as the key. They do not read the meaning. The prefill time and the cache load time depend on the number of tokens, not on their content.
My past life... here