QKV:
Transformer
Computation Atlas

Question

01

Xiao Wang · asks

The previous diagram shows a feed-forward network after attention, continuing to process the current token's internal values. I have heard that MoE sends a token to a few selected experts.

How do these experts relate to attention heads? If two experts are selected, do we get two additional sets of historical K/V?

Answer 1

Da Wang · explains

First locate the operation. We are considering the common MoE design that replaces the feed-forward component with an expert network. Attention first retrieves information from the preceding context; the feed-forward component then uses learned parameters to further process the current token's internal values.

A mixture of experts (MoE) provides several feed-forward networks with different parameters, each called an expert. A small scoring and selection module, the router, examines the current token's internal representation, selects a few experts, runs them and merges their outputs. This increases the total parameters available to the model while executing only a few experts per token. Experts and attention heads occupy different parts of a model layer. [62]

FIG 02Heads read history; experts process the result; K/V stays in attention

Given 5 input tokens, track only token 5 in layer 1. The example has 4 query heads, each storing its own K/V, and 8 feed-forward experts, of which 2 are selected per token.

We reuse the magenta dashed boxes from FIG 01: matching numbers mark work that can run in parallel. Group ② must wait for group ①'s results. Unmarked work is not necessarily sequential.

Attention within this layer

On narrow screens, scroll the diagram sideways. The explanation follows below.

Still inside layer 1: continue processing token 5

The MoE feed-forward component

Colored experts are selected for this token; gray experts are not.

Four heads, eight experts and weights 0.7/0.3 are teaching choices; small colored bars represent internal values. Normalization and residual connections are not drawn individually. This example uses ordinary multi-head attention with one K/V set per head. GQA/MQA, discussed later, can share caches across heads; expert selection is a separate setting. [62]

Real models also configure these counts separately. Qwen3-30B-A3B has 128 experts and activates 8 per token, but its attention has 32 query heads and 4 KV heads. Historical state in each layer is therefore stored for 4 KV heads, rather than 8 or 128 experts. [63]

Xiao Wang Where do expert parameters come from? Who teaches the router? Does training data tell it to select experts 2 and 7?

Da Wang: Consider the common setup in which all components train together. Experts, router, attention and output layer start with initial parameters and jointly learn to predict the next token in training text. The data supplies the actual next token, usually not a “correct expert ID.”

Continue with the first five positions. The training text tells us token 6 is A, but the model may read only positions 1–5 when predicting it. Suppose a toy vocabulary contains A, B and C, with predicted probabilities 20%, 50% and 30%. Training directly evaluates the probability assigned to the true A: lower probability incurs a larger penalty. This penalty is the prediction loss. There is no need to sample a token first and mark it right or wrong.

FIG 03The true answer trains the whole model; expert IDs need no manual labels

We expand one MoE operation at position 5. Solid green: forward predictionDashed red: training feedback and parameter updates The actual token 6 scores the prediction; it is not inserted into its preceding context.

On narrow screens, scroll the diagram sideways. The explanation follows below.

First compute with the current parameters

The router selects 2 and 7. Those experts run and their outputs merge; later layers produce probabilities for the three candidate tokens. No parameters have been updated yet.

Feedback comes from the actual next token

The text identifies A as the answer. The model assigns it only 20%, so the training program computes the corresponding loss. It does not treat 2 and 7 as manually labeled correct experts.

Propagate feedback backward through the computed path

Starting from the loss, the program asks how a small increase or decrease in each parameter would affect it. Those directions and rates of change are gradients; computing them backward through the calculation is backpropagation.

Aggregate, then update parameters

The algorithm that changes parameters is the optimizer. It uses a batch's gradients and other information to adjust them; the next batch uses the updated parameters. Training does not permanently assign an expert ID to each piece of data.

The three candidates, 20%/50%/30% and experts 2 and 7 are teaching choices. The diagram shows one path through joint training; real training processes many positions and examples. Red arrows show gradients through the computed path, not a forward prediction peeking at the answer. [65]

Xiao Wang Without a “correct expert” label, how does the router know what to change? Isn't selecting an expert a discrete choice?

Da Wang: Distinguish the selected IDs from the mixing weights. IDs 2 and 7 determine which branches execute; 0.7 and 0.3 participate in numerical computation. If slightly increasing expert 2's mixing weight would reduce the downstream prediction loss, backpropagation can supply that feedback; the opposite effect gives the opposite feedback.

The router computes 0.7 and 0.3 from this input. Training changes the router parameters that produce scores and weights, while also training the selected experts to process inputs. In the Top-k routing used here, choosing the highest-scoring IDs itself is not directly differentiated. Feedback flows through the selected continuous weights and other differentiable operations. Later changes to scores may also change which experts are selected. [65]

Unselected experts are not individually trial-run at this position. The program does not directly observe what replacing an expert would have done. Only experts 2 and 7 receive task gradients through the expert branches shown here. Other experts may still be selected and trained at other positions or on other examples.

Xiao Wang Could the router keep choosing the same experts while the others never learn?

Da Wang: Yes. A batch with 100 positions and 2 experts per position has 200 assignments. If all choose 2 and 7, those two each receive 100 assignments and the other six receive none. Selected experts keep learning while idle ones get fewer opportunities. If experts reside on different devices, some devices may also be busy while others idle. This is expert load imbalance.

Penalize overly concentrated assignments. For example, Qwen3-MoE's public training path can add an auxiliary load-balancing loss to the prediction loss, discouraging persistent concentration on a few experts. It encourages balanced allocation rather than assigning a subject-specific expert to each input. [65]
Adjust selection scores according to utilization. During DeepSeek-V3 training, routing biases decrease for overloaded experts and increase for underused ones. These biases influence selection, while output mixing still uses the original affinity scores. That model also retains a small sequence-level balancing loss. [66]

As experts receive different data, their parameters develop different tendencies. However, “expert 2 handles math and expert 7 handles code” is not a universal predefined division of work. Specializations may overlap or correlate with lexical, syntactic or contextual features. [67]

How does MoE affect caching benefits?

A prefix hit can also avoid recomputing experts for old tokens. The historical K/V needed by later layers is already available, so the computation that produced those states usually need not run again. New tokens still require their own attention and expert computation.
Selecting few experts does not mean storing only those experts' parameters. Expert parameters and request-specific KV state are different data. More expert weights occupying GPU memory can leave less space for KV.
Subtract distribution and aggregation costs from the computation saved. If experts reside on different GPUs, token representations must be transferred and outputs combined. Actual speedup depends jointly on computation, storage and communication costs. [64]

Answer 2

Da Wang · explains

Imagine an assistant that reads code, uses a terminal and fixes errors. The first request supplies project material, the next returns test results and the third asks for another change. The same tool descriptions, project rules and conversation history reappear in successive requests.

The intuition from a single question

One input → one answer

It is easy to focus only on answer quality and output tokens per second.

→

The reality of a multi-round task

Return history → compute repeatedly → verify

Repeated context processing also determines whether the task is affordable.

Compare a model's efficiency on multi-round work using the whole task's total cost, elapsed time and failed retries under the same acceptance criteria. Such tasks repeatedly submit the same project material and conversation history. Whether caches can be reused, and how much computation that avoids, therefore changes the task's overall cost.

Answer 3

Da Wang · explains

When processing a known token, an attention layer generates numerical vectors. Q is the query, K is used to compute matching scores, and V supplies the information retrieved using the resulting weights. The current token also produces its own K/V, which participates alongside earlier tokens' K/V in this layer. [1]

The internal numerical representation used while processing a token is its hidden state. It is not yet the output layer's scores for candidate tokens.

Fix one response throughout this example. Three user-input tokens occupy sequence positions 1–3. The model has already selected the first response token, at position 4. It is now fed back into the model to predict the second response token, at position 5.

FIG 05How is existing K/V reused within one response?

Read downward: existing input → process token 4 → select token 5

Positions 1–3 are user input; 4 and 5 belong to the same response. Input positions already computed Q/K/V. This step reuses their retained K/V, computes token 4's Q/K/V and selects token 5. Numbers are positions in a simplified sequence, not vocabulary IDs; system instructions and message-boundary markers are omitted. [1][11]

Attention(Q, K, V) = softmax(QKᵀ / √d) V

First match the current query against each key, then combine values using the resulting weights.

A call usually first processes its known input as prefill, then generates new tokens step by step in decode. Prefill can batch and parallelize many known positions. During generation, processing the current known token predicts the next token. Caching avoids recomputing old K/V; reading it and computing new content still cost work. [1]

Only one attention layer is expanded. Ordinary Transformer layers each generate their own K/V; attention heads and other dimensions are omitted here. Selecting token 5 does not mean its own K/V has already been computed. If generation continues, token 5 is fed back into the model on the next iteration to produce its corresponding K/V.

Reusing this history in another request also requires a matching prefix and an available compatible cache.

Answer 4

Da Wang · explains

A network that processes pixels, the vision encoder, first transforms the image into several numerical vectors. A connector feeds them into the language model; an intermediate text description is not required. Two arrangements follow: image vectors occupy positions in the main sequence, or the text sequence stays separate and dedicated attention modules read the image. [70]

Distinguish two lengths. The number of elements in one Q vector is a model configuration choice. The number of K/V records it can match depends on what is available for that query. We compare the latter, starting with five text records.

FIG 06With an image, does Q read 9 records, or separately read 5 and 4?

Use 5 text tokens and 4 image vectors in every comparison. Track one query by the last text token, T5, in the relevant attention module. After all layers finish, its result predicts the first response token, R1. Each query shown reads every record it is permitted to access. Special markers, padding and head grouping are omitted; all image records are assumed visible.

Text positions / their K/VImage positions / their K/VTracked T5 and its QEqual-sized cells make record counts comparable

①No image: 5 text positions

Scroll sideways on narrow screens; notes follow the diagram.

②Image in the sequence: 4 image + 5 text positions = 9

Image first, text afterward; cells have the same size as above.

③Separate image memory: 5 text positions plus a distinct image read

The two attention operations are expanded separately; intervening computation is omitted.

One joint allocation across 9 records ≠ separate allocations across 5 and 4 records.

These are different model architectures. The second retains a shorter text sequence but adds dedicated image-reading modules. Image computation and caching have not disappeared.

The counts are teaching choices for comparing data flow and readable records. Real images do not invariably produce four vectors. LLaVA illustrates insertion into the sequence; Mllama illustrates separate image K/V. Each attention layer may have its own K/V; this diagram does not merge all model layers' records into one cache. [70]

Xiao Wang As the response continues, how do the text and image caches grow?

Da Wang: T5's final hidden state goes through the output layer to produce vocabulary scores and select R1. R1's own K/V is usually not computed yet. The next step feeds the selected R1 back into the model, appends its text K/V and uses the resulting computation to predict R2.

Scroll tables sideways on small screens.

Image connection	T5 predicts R1	Feed R1 back to predict R2
Text only	Self-attention: 5 records	Self-attention: 6 records
Image in the main sequence	Self-attention: 9 records 4 image + 5 text	Self-attention: 10 records Still 4 image; text grows to 6
Image read through cross-attention	Text self-attention: 5 records Image cross-attention: 4 records	Text self-attention: 6 records Image cross-attention: still 4 records

Xiao Wang If image K/V stays unchanged, how can describing colors and reading a label retrieve different information?

Da Wang: If the image features are unchanged and the existing image K/V remains usable, those records can be reused. Later text positions generate different Q vectors, giving different matching scores against the same keys and different weights for reading values. Avoiding repeated preparation of image records does not eliminate each step's query, matching and retrieval. [70]

“Text K/V” means K/V generated at text positions, not K/V containing only textual information: preceding cross-attention may already have incorporated image information into those positions' states. With image vectors inserted into the sequence, changing text before an image can also change deep-layer K/V at image positions. Identical image files alone do not prove the whole language-model cache is reusable.

Answer 5

Da Wang · explains

Start with ordinary new input that misses the old cache. It still computes Q/K/V, reads context and traverses the model layers. To understand efficient input processing, first resolve this puzzle: if later positions depend on earlier context, how can positions be computed together?

Temporarily replace 1,000 input tokens with three known tokens, A, B and C. D, E and F will denote generated response tokens. Distinguish values entering a layer, Q/K/V projected in that layer, and values after the layer finishes; they belong to different stages.

FIG 07Prepare each position's Q/K/V, then read preceding context together

Known input A, B and C occupies positions 1, 2 and 3. Only one attention head and the first two layers are expanded. Colored cells represent numerical vectors.

Parallel ① Dashed boxes with matching numbers can run concurrently; arrows still show dependencies. Numbers match only within this diagram.

① Known input: each position has its own starting point

② Attention in this layer: restrict visibility without queuing rows

③ Next layer: previous context is already in each hidden state

This is an ordinary causal Transformer. Multiple positions reading the same K/V records differs from GQA's sharing across heads or YOCO's sharing across layers. The diagram follows the pinned implementation's dependencies: Q/K/V projection → attention → remaining layer operations → next layer. [40]

Why can't future response tokens also be processed in advance?

Compare the two cases. A, B and C are given at the start; during generation, D must first be selected before its embedding can be looked up and fed into the model. Both cases project Q/K/V before computing attention.

FIG 08How do input processing and generation each produce Q/K/V?

The same ordinary causal Transformer and attention head, expanding only layer 1. These are consecutive steps in one conversation: process input A/B/C, select D, then feed D back to predict E.

Parallel ① Dashed boxes with matching numbers can run concurrently; arrows still show dependencies. Numbers match only within this diagram.

Process input first: all three tokens are given

Continue generation: D is this step's new input

The Q/K/V projection rule is the same; which tokens are given is different. Only layer 1 is expanded. Deeper layers project new Q/K/V from that position's preceding-layer output and read their own layer's K/V cache. Input processing uses C's final representation to predict D; generation uses D's to predict E. [40][11]

Xiao WangSo known input has dependencies between layers, but allows parallel work inside a layer?

Da Wang: Yes, specifically multiple positions performing the same step together. A layer first prepares Q/K/V, then computes attention and finishes subsequent operations. Those stages still depend on one another. Within each stage, positions can be batched once the necessary inputs exist. The next layer still needs the preceding layer's outputs.

Xiao WangDoes generating a new token lose even the parallelism within a layer?

Da Wang: No. The generation side is processing position-4 D to select position-5 E. Multiple query heads within a layer can process D concurrently, each using its own Q. Within one head, Q₄'s matches against different keys can run in parallel; feed-forward operations also contain many parallel numerical computations.

Ordinary generation waits for E and F to be determined in sequence. Before E is selected, its embedding is unknown, so E cannot be processed. Compared with known input, there are fewer opportunities to process several new positions of one response together. Parallelism inside the current token's computation remains. [40]

Xiao WangThen does Q/K/V generation ignore history, with only attention depending on historical K/V?

Da Wang: Distinguish what a step reads directly from what its input already contains. Consider C in input A/B/C:

Layer 1: C projects Q/K/V from its own embedding, before attention reads A or B. Its layer-1 attention then reads A/B/C's K/V. After the remaining operations, C's layer-1 output can already contain information from A and B.

Layer 2: C projects new Q/K/V from its own layer-1 output, so those values can indirectly depend on A and B. Their projection does not first need to read A/B's layer-2 K/V or wait for A/B's layer-2 attention outputs.

Not directly reading historical K/V in this layer does not mean being independent of history. History may enter the preceding layer's output and thereby influence this layer's Q/K/V. [40]

The model first processes the whole input. To generate the first response token, many implementations apply the vocabulary output projection only to the final input position's final-layer representation. Earlier input positions still participate in the model computation; their own next-token vocabulary scores need not all be calculated. Logits are the output layer's raw candidate-token scores, before conversion to probabilities. [11]

Return to the original 1,000-token input. D/E/F still denote response tokens, but their full-sequence positions are now 1,001/1,002/1,003 rather than 4/5/6 in the three-token illustration.

FIG 09Input can be batched; response tokens are selected step by step

The same 1,000 known input tokens are followed by D/E/F. Both panels show model layers progressing downward. Red arrows identify sequential dependencies.

Parallel ① Dashed boxes with matching numbers can run concurrently; arrows still show dependencies. Numbers match only within this diagram.

1Input: process several known positions together

Within a layer, multiple positions can perform the same operation together. Q/K/V projections precede attention and subsequent processing; these stages are not all simultaneous. The next layer uses the preceding layer's outputs.

After final representations exist for all input positions, selecting the first response token D usually needs vocabulary scores only for position 1,000.

2Generation: the current token still permits parallel work

The three branches are different query heads at one position. They can run concurrently. A head's Q/key matches and many feed-forward calculations also allow parallel work.

The unknown identity of the next token causes the wait. Only after D traverses all layers and E is selected do we know which embedding to retrieve. E and F cannot be submitted together in advance as known input can.

3Why can Q/K/V depend on history without directly reading it?

Start with the blue path. Layer 1 projects Q/K/V from this token's embedding without first reading earlier tokens' K/V.

Then follow the green arrows downward. Attention reads history and affects this position's layer-1 output. Layer 2 projects Q/K/V from that result, so they can indirectly incorporate preceding context.

No direct read of this layer's historical K/V does not mean independence from history. Deep-layer Q/K/V receives input that has already passed through preceding-layer attention.

This compares ordinary Transformer input processing with sequential generation of one response. Parallel work within a layer does not remove dependencies between stages or layers. Diagram layout does not represent actual elapsed time. [40][11][28]

For example, after a GPU loads a weight tile into faster memory used by computation, a batch can reuse it for several tokens. Sequential generation handles fewer new positions per step while still accessing weights and historical K/V, making data waits more significant. These weight transfers are mainly between GPU memory and compute units; the model is not reloaded from disk at every step. [28]

Batching does not turn 1,000 tokens' computation into one token's computation. It changes how work is scheduled and how per-token overhead is shared. Generation already uses KV caching. It avoids recomputing old positions, but cannot determine future tokens early or eliminate new-step computation and historical reads.

Output also keeps resources occupied. Until a request finishes, its state must remain available, and further generation adds state. Servers can batch decoding across users, but must balance GPU memory capacity against each user's waiting time.

Answer 6

Da Wang · explains

Compare token 3, “apples.” In a standard model's first layer, before reading context, the same token at the same position can have identical K/V. First-layer attention then reads context containing either “like” or “hate,” potentially producing different outputs. Layer 2 projects K/V from those outputs, so its K/V may differ.

FIG 10The same “apples”: why can first-layer K/V match while second-layer K/V differs?

↔ Scroll sideways to view the full diagram

Each panel places three input tokens horizontally and distinguishes layer-1 and layer-2 K/V vertically. With standard positionwise input transformations, identical “apples” tokens and positions can produce identical first-layer K/V. After reading different context, second-layer K/V may differ. [1]

KV_i^(ℓ) = F_ℓ(x₁, …, x_i; model and runtime conditions)

Subscript i is sequence position; superscript (ℓ) is model layer; x_i is token i. F_ℓ denotes the corresponding layer's computational relationship, not the hash function used to look up cache entries.

A general high-layer KV cache keyed only by token_id or (token_id, position) would therefore conflate different contexts. Earlier conditions must also be distinguished.

This does not yet imply storing the whole plaintext prefix inside every key. That is the next question.

Answer 7

Da Wang · explains

Yes. The server derives an ID from the input prefix, model and other conditions, then looks up retained K/V. It does not have to recompute K/V before deciding whether reuse is possible. [2]

FIG 11Identify the prefix first, then use the ID to find its actual cache

Follow purple to distinguish histories, then green to find already-computed data.

↔ Scroll sideways on small screens; explanations follow each panel.

The same “apples” can have different prefix IDs

h denotes a lookup ID. Words are treated as individual teaching tokens; model and runtime conditions are fixed.

vLLM's hash_block_tokens hashes a parent block's hash, current block tokens and extra identity information. It follows the same idea as parent_hash + current_token, generally operating on blocks. [3]

Answer 8

Da Wang · explains

That is possible. Finer entries can reduce recomputation while increasing identification, lookup and management work. Whether they are worthwhile depends on the actual costs on both sides.

Xiao WangWhat work specifically increases when there are more entries?

Da Wang: First estimate the cost of creating entries, then the computation those entries can save on later requests.

FIG 12Sixteen times as many entries: how much extra computation is avoided?

Compare two direct schemes: one entry per token or one entry per 16 tokens. The grouped scheme reuses only complete groups. Both schemes' old caches are available.

↔ Scroll sideways on small screens; numbered explanations follow their panels.

Creation cost: first process the same million-token document

Reuse benefit: two divergence points in a 1,000-token request

The first 992 tokens are collapsed on the left; the last eight are shown individually. This is not proportional to the full sequence length.

Old: reuse retained resultsRedo: recompute identical contentNew: compute the suffix from the divergence

2Divergence at a group boundary

992 is a multiple of 16. Both schemes reuse up to that point and compute the final eight tokens. The additional entries provide no benefit here.

3Divergence inside a group saves seven tokens

Token-level lookup reuses positions 993–999. With groups of 16, those seven do not form a full group and must be recomputed. Token 1,000 differs, so both schemes compute it.

Counting only group-alignment recomputation: zero extra tokens with token-level entries; at most 15 extra tokens per request with groups of 16.

4Repeated use accumulates the benefit

Assume the old cache stays available and every comparison saves the same additional seven tokens. Avoided computation accumulates with repeated use.

The decision depends on whether the actual cost of avoided computation offsets the extra entry-maintenance cost. Entry count alone cannot decide it.

Under these assumptions, 1 MB = 1,000,000 bytes. The upper panel counts only ID storage; the lower counts input-token computations. Neither directly gives total cost or elapsed time.

In a chained hash scheme using the previous ID and current content, each ID depends on its predecessor. The entire chain cannot simply be generated simultaneously on first creation. The relevant vLLM code proceeds group by group, passing each hash to the next group. [13]

Xiao WangOnce IDs exist, what overhead remains beyond table lookup?

Da Wang: The system still tracks cache locations, current users and when data may be reclaimed.

If two requests share a cache, finishing one must not reclaim data the other is still reading. Starting and ending use therefore require corresponding bookkeeping updates.

vLLM's cache manager processes blocks individually: increase usage counts on a hit, decrease them when requests release blocks, and adjust which blocks can be reclaimed. [13]

If physical management units also shrink from 16 tokens to one, these operations become more numerous. Under concurrency, CPU cache-management time may offset the GPU work saved on a few tokens.

However, finer matching need not imply finer physical storage or management units. They can be designed separately.

Xiao WangCan token-level reuse be retained while maintaining fewer entries?

Da Wang: Yes. Better data structures avoid creating an independent entry for every position in advance.

For example, initially represent 1,000 consecutive tokens as one segment. If a later request matches 999, split the record after token 999. That preserves exact reuse of the first 999 without initially creating 1,000 separate tree nodes.

SGLang's cache tree uses this approach. A node can hold a contiguous segment and split when a match ends inside it. Its ordinary token-matching path supports single-token precision. The split reorganizes cache records; existing K/V remains usable. [5]

The practical comparison is therefore how much additional computation finer reuse saves versus the additional lookup, management and storage costs. Measure total time, memory and service capacity on the same request workload to choose between schemes.

Answer 9

Da Wang · explains

Think of a cache boundary or restore point. A “cutoff point” misleadingly suggests later content is excluded from the model.

Putting cache_control on a content block marks the prefix from the beginning of the effective prompt through the end of that block as a caching candidate. Claude documents the order as tools → system → messages. [4]

FIG 13A cache point restores the prefix “through here”

The purple bracket spans from the start to the marker. Subsequent content still enters the model.

↔ Scroll sideways on small screens; the marker is explained below.

At the marker, distinguish model computation from service-level caching

The marked content block is included in the cached prefix. An API content block is different from a physical block storing K/V on a GPU. [4]

Several cache points can reflect different stability levels: tools and system instructions rarely change, project material changes occasionally and conversation grows continually. If a longer entry becomes unusable, a shorter one may remain valid. All still mean “from the start through this point”; matching text later in the request does not automatically become context-independent cache material.

Answer 10

Da Wang · explains

Fix the scenario: A is the entire document; B, C and D are different questions. Every request begins with A and replaces the question afterward.

Distinguish global K/V from local K/V first

Some models maintain both kinds of KV cache, including the DeepSeek V4/V4.1 discussed here. “Global” and “local” describe their context coverage.

Global K/V: older content remains readable

Records cover the processed long context, allowing the model to revisit earlier parts or the beginning of a report. They may be compressed or shared across model layers. “Global” does not require reading every record at every step.

Local K/V: recent state retained separately by each layer

Each layer retains the K/V it computed for recent tokens to read nearby context. This range is its local window. As new tokens are processed, new records enter and the oldest records leave the current window.

Reading within this recent range is sliding-window attention (SWA). A layer using both paths reads its local window and may select older information from global records, combining long-range information with layer-specific recent state. [12][14]

Layers may retain the same token range, but their local K/V comes from their own computations. Shared global records are also different data. Slicing the end of the global cache cannot simply substitute for every layer's local state. [12][14]

FIG 14The API documentation example: which request first reuses the document?

↔ Scroll sideways to view the full diagram

This is the official API documentation's A+B, A+C, A+D example, assuming no other usable prefix and A saved before request 3. It is neither a guarantee for all live requests nor a measurement in this edition. See the next diagram for global records and local restoration state. [6][14]

If A was computed already, why might C still be unable to continue from it?

The API documentation explicitly relates prefix-retention rules to SWA. The crucial distinction is that recognizing the same prefix does not prove all state needed to continue from its endpoint still exists. [6]

Use a teaching model with a local window of four tokens. Document A occupies tokens 1–4; the old question B occupies 5–8. After processing A+B, we want to replace the question at the end of A with C.

FIG 15Global records remain while local state may be missing

Two states at one moment: the previous request reached token 8; the new request wants a different continuation after token 4.

↔ Scroll sideways on small screens; explanations follow each panel.

Follow the numbers: retain, evict, restore

Numbers are token positions. Only one four-token local window is expanded; model layers have different local states.

Section 3.2.1 of the V4.1 report reviews the earlier V4 deployment: global K/V was retained throughout, while local-window snapshots were mainly kept at input and output endpoints. An intermediate prefix endpoint need not have a local snapshot. [14]

Combining the report with the API example suggests this mechanism: local restoration state at A's end was initially missing, requiring recomputation; once the common prefix was identified and stored, later requests could continue from it. This is a mechanism-based explanation; the API does not disclose every internal live-service step.

How does V4.1 change this constraint?

The bounded-replay method in §3.2.2 reconstructs missing local state by recomputing only a short suffix when global K/V is available. This reduces reliance on long-lived local snapshots. However, those tokens may originally have depended on earlier computation results; without those results, reconstructed state need not exactly match the original. [14]

Answer 11

Da Wang · explains

A cache need not permanently occupy GPU memory. The diagram connects storage, retrieval, reads and independent continuations of the same K/V.

FIG 16The same K/V: storage, reads and separate continuations

Follow three aspects of already-computed K/V while a user leaves and returns.

1Idle state can leave GPU memory

Possible temporary storageHost RAM · SSD
or another machine

Retrieve →← Offload

GPU memoryReady for computation to read

2Once in GPU memory, locate records through a table

Each page here holds K/V for four tokens.

Tokens 1–4First K/V page

GPU block 7This page physically resides here

Tokens 5–8Second K/V page

GPU block 2This page physically resides here

3Separate a writable tail page before two answers diverge

Tail page shared by two responses

Existing K/VFree slot

Separate before appending ↓

Response A

Existing K/VA's new K/V

Response B

Existing K/VB's new K/V

Changing storage locations does not reorder historical token positions. Sharing must not overwrite data another response still needs. [25][47]

Does “static” in StaticCache mean the content stops changing?

No. StaticCache reserves fixed capacity in advance, while new K/V is still written and valid length advances. DynamicCache expands storage as input grows. Fixed capacity works more naturally with fixed-shape compilation but may reserve unused space. This is a separate choice from a sliding window that retains only recent history. [41]

Answer 12

Da Wang · explains

The problem: two large temporary tables beyond K/V

Suppose the model processes 10,000 tokens at once. Consider one Q/K/V set in one layer. A separated implementation does the following:

Compute matching scores. Every input Q scores the permitted keys according to the model's rule. Stacking all queries produces a score table.
Convert scores to retrieval weights. Exponentiate each query's scores and divide by their total so the row sums to one. This is softmax; masked future positions are excluded.
Ordinary Transformers already do this. The index vectors discussed later first select records; selected records are still scored again and normalized with softmax.
Weight the contributions from V. Multiply each value vector by its weight, then add corresponding coordinates across records to form each output coordinate.

If a full square table is materialized in GPU memory, 10,000 × 10,000 is 100 million numbers. At two bytes each, one table takes 200 MB; retaining both scores and weights takes 400 MB. This example assumes full-square storage, including slots for masked future positions. It counts only one Q/K/V set in one layer, with decimal MB.

Capacity is only part of the problem; moving data between two places also costs time. GPU memory holds large datasets. Computation loads relevant portions into fast on-chip storage, such as registers and SRAM, for the compute units.

If scoring, softmax and V aggregation are separate operations, intermediate results may be written to GPU memory and read back on-chip by the next operation. The arrows below expose those transfers.

FlashAttention: finish a tile's consecutive operations on-chip

FlashAttention is an attention computation method. It loads small Q/K/V tiles on-chip and immediately uses temporary scores in subsequent computation, avoiding full score and weight tables in GPU memory. Compare the two paths to see the savings.

FIG 17Which GPU-memory round trips does FlashAttention avoid?

Both panels place GPU memory on the left and on-chip computation/storage on the right. Orange marks the full temporary tables retained by an ordinary separated implementation. Data moves in portions; the whole table is not loaded into on-chip memory at once.

Separated implementation: write a table to GPU memory, then read it back

GPU memoryInput and result storageOn the GPU chipCompute units and fast storage

Q、K

Read in

① Compute scoresQ and K
Matching scores

Full score table

Write out

The score table just produced

Read back

② softmaxConvert scores
to retrieval weights

Full weight table

Write out

Weights + V

Read back

③ Combine VMultiply by weights
Add corresponding coordinates

Attention output

Write out

FlashAttention: load a tile and continue its computation on-chip

GPU memoryInputs and outputs are still storedOn the GPU chipRegisters and on-chip SRAM

Q、K、VRetained in GPU memory
Read in tiles

Load a tile

Compute the current tile here

Q/K matching scores

Exponentiate; update accumulators

Multiply V; accumulate contributions

Keep accumulators hereLoad the next K/V tile,
repeat the computation ↻

When all needed tiles finishDivide to produce output

Attention output

Write out

The lower path tracks a forward attention pass for queries processing K/V in tiles. Computation and data movement are rearranged; the participating attention records are not reduced. [49]

Optional: how can tiled softmax produce the same result?

Reduce the example to three input tokens and track token 3's Q. Suppose exponentiated scores are 1, 1 and 3, with V values 0, 0 and 10. Each V is a scalar for this example.

Delay normalization and keep two accumulators: the sum of exponentials and the sum of exponential-weighted V:

First tile, records 1–2: exponential sum 1 + 1 = 2; V contribution 1 × 0 + 1 × 0 = 0.
Second tile, record 3: update them to 2 + 3 = 5 and 0 + 3 × 10 = 30.
After all tiles: divide once: 30 ÷ 5 = 6, the same as combining V using weights 1/5, 1/5, 3/5.

Do not simply average independently normalized tile outputs. The first tile gives 0 and the second 10; averaging gives 5 and loses their proper relative weights.

What if exponentials become too large?

Large scores can make exponentials overflow. Subtract the largest score seen so far before exponentiating. This rescales all exponentials by the same factor. V contributions receive the same factor, leaving the final ratio unchanged.

If a later tile introduces a larger score, rescale previous accumulators too. In this example, divide all exponential terms by three when tile 2 arrives: the earlier 2 becomes 2/3, the third record's 3 becomes 1, and weighted V totals 10. Then 10 ÷ (2/3 + 1) = 6, still the same result.

Implementations therefore retain the maximum score, sum of exponentials and accumulated V contribution per query, update them per tile, and divide at the end. The mathematics is equivalent; finite-precision arithmetic can still introduce rounding differences when operation order changes. [49]

Optional: the accumulator update equations

m is the maximum processed matching score, ℓ the sum of exponentials after subtracting it, and z their weighted V sum. New-tile scores are sᵢ with values vᵢ:

m′ = max(m, max s)
α = exp(m − m′)
ℓ′ = αℓ + Σ exp(sᵢ − m′)
z′ = αz + Σ exp(sᵢ − m′)vᵢ

α rescales old accumulators to the new maximum. After all tiles, output z/ℓ. Initialize m = −∞, ℓ = 0 and z = 0. Include only readable positions; causally masked future tokens do not contribute.

What is eliminated, and what remains?

It avoids storing two full temporary tables and the GPU-memory reads/writes around them. Reduced movement can improve speed; it does not delete historical K/V or skip Q/K matches that should participate.

Method	Main saving	What remains necessary
Prefix caching	Avoid regenerating K/V for identical input	New queries still read history and compute
PagedAttention	Avoid one large contiguous allocation; reduce wasted space	The K/V records themselves still occupy memory
FlashAttention	Avoid full score and weight tables in GPU memory	Still read K/V participating in this attention operation

These techniques can be combined. FlashAttention's public cache interface accepts page tables, but support for combinations of paging, quantization and window attention must be checked for each implementation. [48]

Answer 13

Da Wang · explains

The number of query heads need not equal the number of stored K/V groups. Fix one model layer and calculate per token. In this example, eight query heads share K/V in groups of four, giving two KV groups.

If each K and V has four numbers and each number takes two bytes, one pair occupies (4 + 4) × 2 = 16 bytes. Two groups total 32 bytes per token. Sum over retained positions and the actual per-layer shapes for the basic cache payload.

One KV group per query head is multi-head attention (MHA). One group shared by all query heads is multi-query attention (MQA). Between them, groups of query heads share K/V in grouped-query attention (GQA). This shares across heads, not requests or layers. [39]

Xiao WangDoes “several query heads share K/V” refer to sharing within a head, queries at different token positions reusing history, or different heads' queries for the same token sharing K/V?

Da Wang: Here it means different query heads within the same layer share K/V. Fix one token: it produces distinct Q vectors in different heads, which may read the same set of historical K/V.

With five input tokens, consider token 5 in this layer. It produces eight Q vectors, one per query head. Heads 1–4 read KV group A; heads 5–8 read group B. Each group still retains separate K/V for every position 1–5.

For a reminder of layers and heads, return to FIG 01: multiple heads per layer, with results passed between layers.

FIG 18Eight query heads can retain only two K/V groups

Fix one layer and five input tokens. All eight Q vectors belong to token 5, produced by heads 1–8. Both cache groups A and B retain distinct records for tokens 1–5.

Parallel ① Dashed boxes with matching numbers can run concurrently; arrows still show dependencies. Numbers match only within this diagram.

Query heads 1–4

Parallel ①

H 1Use their own QCompute attention

Parallel ①

H 2Use their own QCompute attention

Parallel ①

H 3Use their own QCompute attention

Parallel ①

H 4Use their own QCompute attention

Share historical K/V group A
Tokens 1–5 each retain their own K and V

Query heads 5–8

Parallel ①

H 5Use their own QCompute attention

Parallel ①

H 6Use their own QCompute attention

Parallel ①

H 7Use their own QCompute attention

Parallel ①

H 8Use their own QCompute attention

Share historical K/V group B
Tokens 1–5 each retain their own K and V

All eight heads are marked Parallel ①. Once this layer's Q/K/V is ready, these attention operations can run concurrently. A and B are cache groups, not sequential execution batches.

Shared K/V does not require identical Q. Heads query their group's records with their own Q, so scores, softmax weights and outputs may differ.

GQA groups query heads within a layer to share historical K/V, while retaining a separate record for each token. [39][40]

One “K/V group” is a history indexed by token position. Group A still contains K/V for token 1, token 2 and so on through token 5. Heads 1–4 share those records; the five tokens have not been merged into one K/V pair.

Xiao WangWhat about queries from different tokens within the same head reading the same history?

Da Wang: That happens too. Head 1's Q at token 4 and its Q at token 5 can both read token 2's retained K/V. This reuse along the sequence already exists in ordinary MHA. GQA and MQA additionally share history across query heads. [39][40]

For the same five tokens and eight query heads, MHA stores eight K/V pairs per token, 40 pairs total. Two-group GQA stores 10 pairs; MQA stores 5 pairs. The KV-head count falls; separate token records remain.

Teaching configuration: 32 layers, 32,768 tokens, head dimension 128, 2 bytes per number	KV heads	Basic K/V payload
MHA: 32 query heads	32	16 GiB
GQA: still 32 query heads	8	4 GiB
MQA: still 32 query heads	1	0.5 GiB

Only K/V numbers are counted, excluding page tables, alignment and temporary workspaces. 1 GiB = 2³⁰ bytes. These are different architecture choices; deleting KV heads from an already-trained model does not preserve the same outputs. [39]

MHA/GQA/MQA determine how heads within a layer use K/V; YOCO changes connections between the earlier and later model layers. Assume eight query heads per layer throughout, using four model layers for YOCO.

Method	Who shares with whom?	What is saved, and what remains?
MHA Multi-head attention	Each query head has its own K/V group. Eight query heads correspond to eight KV groups.	Independent K/V per head provides the capacity baseline for the next two rows. Already-computed history can still be reused by later tokens.
GQA Grouped-query attention	Heads share by group within one layer. For example, heads 1–4 read A and 5–8 read B: two KV groups.	This reduces K/V from eight groups to two, lowering projection, storage and retrieval costs. There are still eight query heads, each computing its own scores, retrieval weights and result.
MQA Multi-query attention	All query heads in one layer share. Eight query heads read one KV group.	K/V falls from eight groups to one. Heads share K/V, but each still has its own Q and attention computation. [39]
YOCO Model architecture	Several later model layers share global K/V. For example, layer 2's output prepares global K/V read by layers 3 and 4. Earlier layers retain their own local or recurrent state.	Later layers no longer each store a complete global history. During prefill, input positions not needed for the next-token prediction can skip the later half. During sequential generation, the current token still passes through both halves; later layers retain their own query, attention and feed-forward computations. [7][8]

YOCO can be combined with GQA. In this four-layer example, layers 3 and 4 may jointly read global KV groups A and B. Each layer's eight query heads then split into groups of four reading A or B. One global cache here contains two KV groups; it does not mean one KV head. [7][8]

Answer 14

Da Wang · explains

Representing a number with fewer bits is quantization. Think of replacing a fine scale with fewer discrete levels: store the level index plus information about the group's magnitude. Reconstructed values are usually approximations.

If one step represents 0.1, an original value of 0.32 can be stored as level 3, reconstructing 3 × 0.1 = 0.3. Fewer bits lose some precision. The scale describing “0.1 per step” must also be retained.

FIG 19Why does shrinking 32 bytes to 12 not imply an equivalent speedup?

Count actual stored bytes, then follow how compressed values participate in attention.

The same 16 numbers

BF16: 2 bytes per number32 bytes

4-bit data8 bytes＋Scale4 bytes＝12 bytes

Scales also occupy space. Here, 16 numbers share a four-byte scale indicating the step size. Level 3 with step 0.1 reconstructs as 0.3.

The reconstruction path also affects speed

Reconstruct the whole range first

Read low-bit data → reconstruct higher precision → write GPU memory → attention reads it

Adds storage and transfers for reconstructed data.

Reconstruct while reading

Read a tile → reconstruct and compute near the execution units → next tile

Avoids whole-range round trips but still requires unpacking and scaling.

Byte counts use Appendix C's teaching INT4 format. Other formats may have different scales and execution paths. [43][44]

Xiao WangIf the cache shrinks to one quarter, does generation become four times faster?

Da Wang: No. Reduced transfers can improve latency when cache-read bandwidth was limiting, the kernel effectively uses the compressed path and added overhead is small. For short context or ample memory, quantization may be slower. Include reconstruction workspaces in peak-memory measurements. [41]

Xiao WangK and V are both numerical vectors. Why quantize them differently?

Da Wang: K errors first change matching scores, then softmax retrieval proportions. V errors directly affect the weighted sum. Different quantization schemes may therefore suit K and V; suitability depends on the model and task. [42]

If all low-bit data is reconstructed into higher-precision tensors and retained, compressed-file size alone cannot predict runtime memory. Appendix C records actual packed-data and scale bytes separately.

Answer 15

Da Wang · explains

YOCO is a language-model architecture. Several later layers read global K/V produced by the earlier half, reducing duplicated storage and letting some input positions skip later computation. The tradeoff is that those later layers no longer independently produce global history; the model must be trained with this architecture. [7][8]

The paper divides the model into an earlier Self-decoder and later Cross-decoder. The first forms contextual representations and shared global K/V; each later layer reads this memory with its own query. [7]

Xiao WangDo the later layers sharing global K/V also have private local K/V?

Da Wang: In original YOCO, the later layers do not additionally retain private per-layer local K/V. They query the global K/V provided by the earlier half with their own Q.

The earlier half still retains its state. With sliding windows, each layer stores recent-token K/V. A recurrent variant instead retains an evolving state, which should not simply be called “local K/V for recent tokens.” [7][8]

No separate local cache does not mean no access to recent tokens. If the shared global cache covers tokens 1–1,001, recent positions 990–1,001 are included and remain readable by later layers.

Xiao WangDuring decoding, later layers still have their own queries, attention and feed-forward operations. Doesn't that create local K/V of their own?

Da Wang: Having a Q and further computation does not imply projecting private K/V. Ordinary Transformers usually project Q/K/V in each layer. YOCO changes that arrangement: later layers project Q, while the earlier half supplies K/V. Attention uses all three, but they need not all originate in the current layer.

Use the four-layer example, first processing 1,000 known inputs, then generating the response token by token. The main difference between these phases is which positions must traverse the later half.

FIG 20Original YOCO: what can prefill skip, and what must decoding still compute?

① First process 1,000 known tokens: prefill

The shared cache does not yet exist. To begin the response, only the final input position's prediction is needed to select the first response token.

↔ Scroll sideways; gray boxes mark positions that can skip the later half.

② Feed selected token 1,001 back to predict 1,002: decode

The newly fed-back token 1,001 still traverses all four layers. The earlier half projects its K/V; later layers query the updated shared cache with their own Q and continue processing.

Blue: token 1,001's intermediate computationGreen: K/V writes and reads

↔ Scroll sideways; notes on the right correspond to layers on the left.

Both diagrams use the same four-layer teaching model. Prefill can skip the later half at positions 1–999 because only the final position's prediction is needed. All input positions' global K/V must still be prepared. In decode, each fed-back token traverses both halves. Normalization and residual connections are omitted. [7][8]

The reference implementation's later attention module has q_proj for Q and out_proj for its output; K and V are supplied externally. [8]

DeepSeek V4.1 Flash, discussed later, uses a different combination: shared global K/V plus private per-layer local-window K/V. Its reference code has window_kv_cache to generate and retain local records. Local caches exist because the architecture explicitly provides that path, not merely because later layers still execute attention and feed-forward operations. [12]

FIG 21How much space do two retention schemes use for the same input?

↔ Both bars start at zero and use the same linear scale

Teaching configuration: eight KV heads, dimension 128, separate K and V, two bytes per number. The shared scheme additionally retains local windows for the first 20 layers. Decimal units: 1 GB = 1,000 MB; displayed values are rounded. Only the specified K/V is counted, excluding weights, activations, workspace and metadata. A volume ratio is not a speed ratio; no real model was run.

As context grows, fixed-size local windows contribute a smaller proportion.

Both halves remain causal: a position can use only permitted history, and subsequent generated tokens traverse both halves. Describing the earlier half as encoder-like does not make it bidirectional or allow future reads. [7]

Answer 16

Da Wang · explains

FIG 22Which token positions can skip computation in which layers?

Process 1,000 known inputs to generate the first response token. The model has four computational layers. Columns denote token order; rows denote model layers.

“Compute” in a cell: retrieve information in this layer, then process the result through feed-forward operations.

“Project K/V” in a cell: generate this layer's cache from the previous layer's result, not retrieve an old cache.

Each layer is a complete block; input transformation and final output projection are separate. Four layers are used only for illustration.

↔ Scroll sideways; explanations follow each diagram.

Ordinary Transformer

YOCO

2Green: the K/V needed by later layers is ready early

All 1,000 input positions finish layer 2 and project their individual K/V records, jointly shared by both later layers.

Later layers no longer need earlier tokens to continue computing deeper historical K/V. Positions 1–999 can skip layers 3 and 4. [8]

3Blue: token 1,000 still traverses all four layers

Blue carries this token's intermediate representation. In layers 3 and 4 it queries the whole shared K/V cache and finishes each layer's feed-forward processing.

Both illustrated models send only the final input position's internal representation to the output layer, producing vocabulary scores and selecting token 1,001, the response's first token. Positions 1–999 need not each produce prediction scores. [11]

Layers 3 and 4 still exist. The earlier input positions' computations in those layers are omitted.

Next: continue the same response

The first two diagrams share the same axes. Positions 1–998 are only visually collapsed; each retains distinct computation results and K/V. The green cache row is not another model layer. For old positions at layer 4 of the ordinary model, only the necessary K/V projection is shown. This comparison concerns generation following known input. If training or scoring requires predictions at multiple positions, their later-half computation cannot be skipped this way.

Answer 17

Da Wang · explains

Some architectures define K/V as fixed linear transformations of a shorter vector generated for each historical position. That shorter vector is a latent representation.

When the expansion rule allows it, work can move to the query side: transform the current query once, then match many compact records directly. Consider a three-element K generated from two stored numbers.

FIG 23Store one fewer number; move some work to the query

This example shows unscaled matching under a linear relationship, without positional rotation.

Model rule: two numbers generate a three-element K

Retain only [2, 1]; when expanding, the third element is the sum of the first two:K = [2, 1, 3]

The same Q = [1, 0, 2], two equivalent calculations

Expand each historical K, then match

1×2 ＋ 0×1 ＋ 2×3

Matching score: 8

Transform the query once, then match compact records

The query becomes [1 + 2, 0 + 2] = [3, 2].

3×2 ＋ 2×1

Matching score: still 8

One transformed query can be reused across many compact records, avoiding expansion of every historical K for this query.

This teaching example assumes the specified model structure. It does not show that arbitrary existing K vectors can be shortened losslessly.

If V also comes from a linear transformation of compact records, first form their weighted sum, then expand that sum. The architecture must support these relationships; positional rotation and nonlinear operations cannot be reordered arbitrarily.

There is still one record per historical position, just shorter. Combining history into fixed-size state is a different architecture. The next question demonstrates writing and reading a small matrix; restoration conditions appear in Appendix A, Exercise 3.

Answer 18

Da Wang · explains

A model can use a different attention rule. In one classic form of linear attention, prescribed functions transform Q and K before their similarity is computed. Its algebra lets historical contributions accumulate, so a new query directly reads accumulated state. “Linear” here means that, for fixed dimensions, the main attention computation over a sequence grows linearly with its length. [68]

Fix one head in one layer. With two-element transformed K and two-element V, retain a 2×2 matrix S and a two-element vector z. S accumulates outer products of transformed K with V, allowing Q to calculate the score-weighted V sum. z accumulates transformed K, letting the same Q calculate total matching score. Dividing the first by the second converts scores to retrieval weights summing to one. We give the transformed values directly to show the state updates and reads.

FIG 24Two writes, one query: six numbers replace an expanding record list

Two input tokens are given. Each position first incorporates its K/V into state, then reads it with Q. Initially S and z are zero. Cells are numerical components; matrix rows and columns are not token positions. The two panels show the same state at different times.

Position 1

Transformed K12

V20

Multiply each K component by each V component

1×21×02×22×0

Add those four products to S; add transformed K to z

S after this update2040z after this update12

Position 2

Transformed K21

V03

Multiply each K component by each V component

2×02×31×01×3

Add those four products to S; add transformed K to z

S after this update2643z after this update33

Position-2 query: transformed Q = [1, 2]

Read the weighted V sum from S and total score from z, using the same Q.

Read weighted sum from S

12×2643→1012

First coordinate: 1×2 + 2×4 = 10
Second coordinate: 1×6 + 2×3 = 12

Compute total score from z

12·33→9

1×3 + 2×3 = 9
z is [3, 3]; 9 is this Q's dot product with z.

Why divide by 9?

Expand the two records to interpret that 9. This only explains the same calculation; recurrent reads need not retain or rescan the two keys.

Record 1: match Q with K1×1＋2×2＝5Corresponding V = [2, 0]

Record 2: match Q with K1×2＋2×1＝4Corresponding V = [0, 3]

Total score: 5 + 4 = 9

Nine weight units: five for record 1, four for record 2

Record 1: 5/9 ≈ 55.6%Record 2: 4/9 ≈ 44.4%

Weighted read: 5/9 × [2, 0] + 4/9 × [0, 3]

Accumulate raw-score contributions, then divide by 9[10/9, 12/9] ≈ [1.11, 1.33]Equivalent to mixing the two V vectors with weights 5/9 and 4/9, then continuing computation

These nonnegative matching scores from transformed Q and K are normalized directly, without another softmax. Ordinary softmax attention also has weights summing to one, but first exponentiates the matching scores.

This teaching example follows the specified linear-attention rule and omits the small constant commonly added to avoid division by zero. State size depends on head count and vector dimensions; one head is shown. Constant size does not guarantee less memory than a few explicit records on short inputs. [68][69]

Xiao Wang What exactly is saved by merging history, and what can be lost?

Da Wang: Compare this linear-attention rule with ordinary full attention retaining per-position K/V. Hold layer count, heads and dimensions fixed and examine inference-time attention state and computation.

Scroll tables sideways on small screens.

Aspect	Benefit	Cost and conditions
How does retained history grow?	Recurrence retains only current S/z, independent of token count. This head need not append records indefinitely in a long conversation.	Contributions are mixed; an original-position record cannot be retrieved independently as in a full KV cache. Fixed-size, finite-precision state cannot guarantee lossless retention of every detail from arbitrarily long history.
How much is read and computed per generated step?	A new Q reads fixed-size S/z rather than scanning all historical records. With fixed dimensions, attention computation over the input grows linearly with length.	Matrix updates and feature transforms still cost work. For short inputs, a fixed matrix need not be smaller or faster than a few K/V records. Actual speed depends on implementation and hardware, not the word “linear.”
Can it directly replace an existing model?	The same linear-attention rule supports either explicit summation or S/z recurrence, avoiding recomputation of old records for each query.	It changes the ordinary softmax-attention rule and needs corresponding model design and training. Swapping only a cache container cannot guarantee unchanged answers or capabilities.

State shape, updates and reads can be checked in the authors' implementation; computational complexity is discussed in paper §§3.2–3.4. [68][69]

Returning to an earlier position requires old state or enough information to undo updates. Current S/z does not automatically preserve earlier versions. Save snapshots, recompute from an earlier point, or, for the additive example, retain contributions to undo the relevant updates. If speculative decoding tries four steps and accepts two, simply “delete the last two K/V records” is not the right operation here. See state rollback and the storage comparisons in Appendix A, Exercise 3.

Positions are shown sequentially to expose updates. Known input can still use batching and prefix accumulation; it need not run the entire model serially for every position. A hybrid model with full-history attention layers need not have constant-size total state.

Answer 19

Da Wang · explains

FIG 25One vector serves K and V: eliminate two separately stored payloads per record

Track historical token 3's record m₃: stored once, used by two different computations.

↔ Scroll sideways on small screens; explanations follow each panel.

One stored object feeds two paths

Sharing a vector between K and V is separate from sharing a cache across model layers. See the next question for positional rotation and its inverse.

Answer 20

Da Wang · explains

“Rotated K plus V without this layer's rotation” is a common cache format, but reading it does not first inverse-rotate K. The current Q is rotated for its own position and matched directly with cached K. The inverse rotation discussed here happens later: first form a weighted sum of retrieved information, then transform that new result.

Clarify “sharing” first: heads or layers sharing a KV group does not mean K and V are the same vector. This question concerns the latter design from the previous question. Rotary positional embedding, RoPE, makes position affect Q/K matching; this special architecture also uses the rotated vector as V.

Basic RoPE equations and relative position

Fix one attention head. Let (a, b) be any pair of Q or K coordinates and p the token's sequence position. Basic RoPE rotates that pair as follows:

[\begin{matrix} a^{'} \\ b^{'} \end{matrix}] = [\begin{matrix} \cos (p θ) & - \sin (p θ) \\ \sin (p θ) & \cos (p θ) \end{matrix}] [\begin{matrix} a \\ b \end{matrix}]

θ is the angular increment per position for this coordinate pair, in radians; different pairs can use different θ. p is sequence position, not vocabulary ID or the cache's GPU address.

Write the whole-vector rotation as R(p). Query position is m, key position n, and q_m, k_n are unrotated vectors. T denotes transpose. With fixed rotation frequencies:

{(R (m) q_{m})}^{T} (R (n) k_{n}) = q_{m}^{T} R (n - m) k_{n}

Matching separately rotated vectors combines the rotations into relative position n−m. Moving cache storage does not change those positions. Changing preceding context can also change the unrotated vectors, so adjusting angles alone does not establish cache compatibility. [71]

FIG 26From cache to output: which numbers does inverse RoPE transform?

First distinguish two cache formats, then follow one vector serving K and V to calculate a new retrieved result.

1Cache format: separate K/V or one vector serving both?

Ordinary RoPE: K and V stored separately

KRotated at its historical position

VNo RoPE rotation from this layer

Rotate the current Q and match cached K directly. Read and combine V; there is no later output inverse rotation of the kind shown below. [11]

This architecture: one vector serves K and V

Store one rotated vector

As K: matching scoresAs V: retrieved information

The current Q still matches cached K directly. Because the values read as V also carry rotation, the newly retrieved result needs further transformation. [12]

Matching in both cases:

Rotate Q at its current positionMatch cached K directlyObtain record weights

Do not inverse-rotate K first. Relative Q/K rotation already appears in the match.

2Retrieve: weight each record, then add corresponding coordinates

Follow only the shared-vector architecture now. Process token 10 and, for illustration, read only records 8 and 9, each represented by two numbers. The table already contains rotated cached values.

Q and K first multiply and sum corresponding coordinates: dot-product scores. After required scaling and masking, softmax converts these into retrieval weights. Softmax is an attention operation, not an extra model layer. Given weights 0.75 and 0.25, examine their effect on V.

Record 8Cached values

[2, 0]

× 0.75Multiply both numbers by

[1.5, 0]2 × 0.75 = 1.5
0 × 0.75 = 0

Record 9Cached values

[0, 4]

× 0.25Multiply both numbers by

[0, 1]0 × 0.25 = 0
4 × 0.25 = 1

Add both contributions: blue with blue, green with green

Coordinate 1 of each vector1.5 + 0 = 1.5

Coordinate 2 of each vector0 + 1 = 1

The newly retrieved vector[1.5, 1]

This is weighted aggregation. Contributions from two records become one new two-element vector. Their caches are neither concatenated into four elements nor overwritten.

3Inverse rotation comes last, acting on the new vector

Continue the teaching rule of 10° per position. The current query is position 10. Rotate the two newly aggregated numbers by −100°, rather than modifying cached K.

Newly aggregated result[1.5, 1]

Inverse RoPERotate by −100°

New rotated result, approximately[0.724, −1.651]Continue downstream computation

−100° is the additional rotation, not the result's final orientation. Rotating the new sum once equals rotating each weighted contribution this way and then adding. Contributions are expressed relative to the current query position.

The original cache remains:Record 8 [2, 0]Record 9 [0, 4]Available for the next step's reads.

Vector values, weights and 10° increments are teaching choices; the two numbers are not token indices. Real implementations have more coordinates and records and inverse-rotate only the RoPE portion of the output. This does not recover an ordinary sum of unrotated V vectors. [12][36]

“V has no position information” should be narrowed to “V did not receive this layer's RoPE rotation.” V comes from intermediate computation and may already include position-related information from earlier layers. No rotation here does not mean no positional information at all.

Code comparison: cache writes, weighted summation and inverse output rotation

A common Llama implementation calls apply_rotary_pos_emb on Q/K before updating the cache, storing rotated K and V without that rotation. [11]

The shared-vector implementation first rotates and stores kv. sparse_attn uses it both to score Q and to aggregate weighted coordinates. Only then does apply_rotary_emb(..., inverse=True) transform the output. [12]

matching_scores = dot(current_rotated_Q, cached_K)
retrieval_weights = softmax(matching_scores)
retrieved = sum(weight[i] * cached_vector[i] for each record i)
next_input = inverse_rotate_RoPE_part(retrieved, current_position)

The model is designed and trained this way; merging arbitrary K/V does not reproduce it. num_key_value_heads = 1 indicates a small KV-head count, not that K and V are one vector.

Answer 21

Da Wang · explains

Because history being available differs from history participating in this particular computation.

Assuming the conversation is neither truncated nor replaced by a summary, its full submitted history belongs to this context. The model computes K/V for it or reuses retained K/V.

Caching only avoids regenerating historical K/V. Every new token still produces a current Q, searches for historical information and combines the retrieved content.

Full attention includes every permitted historical record in this process. With long history, each step reads and computes over substantial data. Fitting in GPU memory does not guarantee fast reads.

One DeepSeek V4.1 Flash approach uses shorter index vectors to score and select some global K/V. Main attention reads those selected records together with the current layer's recent local-window K/V. [14]

What step does indexing add to ordinary Transformer attention?

Consider one query in one layer using already-prepared K/V. Compare with the earlier softmax explanation: select participating records first, then compute each V contribution.

Comparison	Ordinary Transformer Full attention	DeepSeek V4.1 Flash Index selection + main attention
Which records enter main attention?	All K/V records permitted for this query.	Selected global records plus this layer's recent local-window records.
How are scores calculated?	Main Q directly matches the participating main K records.	Shorter index Q/K first score and select records; main Q/K then rescore the records that actually participate.
How are V contributions determined?	Softmax of main-attention scores supplies weights for combining V.	Again, softmax of main-attention scores supplies retrieval weights. Index scores select records; they are not directly multiplied by V.
What computation remains?	Full attention still scores a record and includes its weighted V contribution even if its eventual weight is tiny.	An unselected record outside the local window does not enter this main-attention operation. Its K/V may remain cached for later queries.

Some DeepSeek layers reuse an earlier layer's selected list but still calculate retrieval weights with their own main Q. Compare ordinary attention with the original Transformer paper ↗, and selection with the pinned report and code. [12][14]

Submitting 100 conversation rounds does not require every generated step to perform the same full attention over every corresponding record. They can remain cached and be selected differently at different steps.

Selection reduces per-step historical work but may omit useful information. Indexing itself costs time and storage, so computing full attention can be cheaper on short history.

Answer 22

Da Wang · explains

What is learned in advance is a rule for processing input. Once your document arrives, the model derives new numerical representations from it and applies the learned transformations.

For example, reduce a record to [2, 1] and define one index coordinate as:

Multiply the first number by 0.6 and the second by −0.2, then add.
2 × 0.6 + 1 × (−0.2) = 1.0

2, 1 comes from this input; 0.6, −0.2 are retained trained parameters. A new record [5, 3] uses the same rule to give 5 × 0.6 + 3 × (−0.2) = 2.4. Content and results change; parameters do not change during ordinary inference.

1.0 or 2.4 is one index-vector coordinate, not the final relevance score. Several parameter sets produce complete index vectors, then historical index K is scored against current index Q. These numerical coefficients are illustrative, not actual model parameters. [12]

Distinguish two meanings of “weights”:

Model parameters: retained values such as 0.6, −0.2 that apply the same transformation to different inputs.
Retrieval weights: values such as 0.75, 0.25 in Question 20, calculated from this query and its history to determine V contributions now.

Explaining how parameters acquire those values requires the training feedback that corrects their judgments. The next question develops that process.

Answer 23

Da Wang · explains

An indexer can have its own training objective to learn score allocation. First distinguish two operations both called masks:

Hide text and predict the original: replace “Xiao Wang lives in Beijing” with “Xiao Wang lives in [MASK]” and predict “Beijing.” This is masked language modeling, used by BERT. [38]
Prevent a position from looking ahead: predict the next token from preceding context and compare with the actual continuation. The causal attention mask forbids future-position reads. It defines visibility, not which historical records are most useful.

V4.1 Flash's language backbone is a causal next-token model. How its selector learns to choose history is a separate training question. [14]

DeepSeek V3.2 gives a concrete public example: compute full main attention, then train a small indexer to imitate its allocation. Main attention reads and combines K/V; the indexer cheaply preselects records. The diagram depicts the initial dense warm-up stage of V3.2 indexer training. [37]

The starting V3.1-Terminus model is already trained before the indexer is added. Main-model parameters are frozen only during indexer warm-up; training resumes afterward. [37]

FIG 27How does training reveal an indexer scoring the wrong records?

V3.2 dense warm-up: freeze the main model temporarily and update only the indexer. The three records and percentages are teaching choices.

Same training example · Same query positionUse history to answer: how much did sales grow this week?

1Main-attention reference2Current indexer allocation

Every full bar represents 100%; each allocation across A/B/C sums to 100%.

ALast week's sales: 80

Reference45%

Indexer10%

BThis week's sales: 100

Reference45%

Indexer20%

CThe office is on floor three3Clearly overallocated

Reference10%

Indexer70%

Convert that discrepancy into parameter-update feedback

KL divergence compares reference and indexer allocations

For this query position, p_i is main attention's reference weight for historical record i; r_i is the indexer's softmax weight. Each distribution sums to one.

D_{KL} (p ∥ r) = \sum_{i} p_{i} \ln (\frac{p_{i}}{r_{i}})

∑_i sums over historical records; ln is the natural logarithm. KL divergence is nonnegative and zero for matching distributions. Its direction here is reference p relative to indexer r; reversing them generally changes the value.

With p = (0.45, 0.45, 0.10) and r = (0.10, 0.20, 0.70), the result is approximately 0.847. [37]

How does this compare with entropy and cross-entropy in training? See the comparison after the diagram.

Compare two allocations

KL divergence measures their discrepancy as a training loss.

Calculate parameter influence

Backpropagate how the loss changes with parameters.

Update indexer parameters

Adjust indexing and scoring coefficients slightly toward the reference allocation.

Repeat computation, comparison and adjustment over many examples. Retain the resulting parameters to score new inputs.

These percentages allocate attention across historical records. They are not next-token vocabulary probabilities or probabilities that records are truly useful. The diagram illustrates V3.2's disclosed training procedure, not undisclosed V4.1 details. [37]

Xiao Wang The indexer imitates main attention using KL divergence. Next-token training often uses cross-entropy, and there is also entropy. Which training objectives suit each?

Start with the training goal, then choose the measure

Da Wang: Sometimes a model learns known answers, sometimes it imitates a probability distribution, and sometimes it should explore more choices. These quantities emphasize different things, but do not define three unrelated task categories. One training procedure can combine them.

Let r_i be the model's probability for candidate i and p_i an existing target probability. Candidates may be vocabulary tokens, classes or historical records, but one comparison must use the same candidate set. Equations use natural logarithms; ∑_i sums candidate contributions.

Scroll tables sideways on small screens.

Quantity and meaning	Suitable training goal	How it is used
Entropy $H (r) = - \sum_{i} r_{i} \ln r_{i}$ Inspect r itself: how dispersed are its choices?	Encourage exploration. In reinforcement learning, for example, a policy should pursue reward without prematurely choosing only one action, retaining opportunities to try alternatives.	Rewarding higher entropy is equivalent to subtracting weighted entropy from a minimized total loss. Low entropy means greater certainty, not greater correctness. A model confidently choosing the wrong answer also has low entropy. [73]
Cross-entropy $CE (p, r) = - \sum_{i} p_{i} \ln r_{i}$ Use target p to assess how much probability r gives desired outcomes.	Learn known answers. Common uses include multi-class classification, next-token pretraining, masked-word prediction and supervised fine-tuning on demonstrations. The target can also be a full distribution rather than one answer.	Minimize cross-entropy to raise target-required probabilities. If A is correct and receives 100% target probability, this example's loss is −ln r_A. No sampled token must first be judged correct or incorrect. [72]
KL divergenceD_KL(p ∥ r)How different are the distributions? Equation above.	Learn a reference distribution. A student imitating a teacher's candidate probabilities is knowledge distillation. Here, the indexer imitates main attention's historical allocation.	Fix reference p and adjust r to reduce D_KL(p ∥ r). It compares complete distributions, not merely agreement on the top choice. Reversing the direction can change the training emphasis. [37][74]

Why can cross-entropy and KL sometimes be interchangeable?

CE (p, r) = H (p) + D_{KL} (p ∥ r)

If target p is fixed and only predictions r change, H(p) stays constant. Under the same probability definitions and weighting, minimizing cross-entropy and KL in this direction gives the same update direction for r. H(p) is the target's entropy, not H(r), used above to encourage exploration. [72]

In the earlier MoE training example, the correct token is A with predicted probability 20%, giving cross-entropy −ln 0.2 ≈ 1.609. The target is “A 100%, others 0%,” with zero entropy, so KL also equals 1.609. If a teacher supplies probabilities across several candidates, cross-entropy can learn that distribution too.

Balanced MoE expert use does not simply mean maximizing entropy. Making every token equally uncertain across experts differs from distributing an entire batch's work sensibly. The earlier Qwen3-MoE implementation adds a dedicated balancing auxiliary loss to the prediction loss. [65]

Where did the main-attention reference come from? The backbone already learned text prediction: predict subsequent tokens from context and adjust parameters, including attention parameters, against the correct text. The indexer uses that learned retrieval allocation rather than manually labeled relevant/irrelevant historical records.

In V3.2's subsequent sparse training, the backbone resumes updates using text-prediction loss, while the indexer learns from allocation differences on selected records. The two training signals are separate. At inference, the indexer selects records before main attention; full main attention need not run first as a teacher for every answer. [37]

“Relevance” here means approximating main attention's historical selection. A high weight does not prove a record is essential to the correct answer. Task outcomes must reveal whether useful information was omitted.

An additional selector-training objective was already explicit in V3.2; it is not a new masked-word task. The V4.1 Flash report describes indexer optimization and gradient aggregation but does not disclose an equally complete indexer-loss definition. Participation in training is established; an identical complete procedure is not. [14]

Answer 24

Da Wang · explains

Neither unselected records 1 and 3–8 nor selected record 2 needs recomputation. Record 2 is read from the old cache. Records 1 and 3–8 remain retained but do not participate in that layer's global main attention.

Record 9 is newly prepared K/V for this step's input. Token 9 was just selected and is only now processed by the model. Its new state must not be conflated with the old cache for positions 1–8.

The query is Q₉ generated at position 9 in the current layer. Selected records affect position 9's subsequent internal values. The model eventually uses that position's computation to select token 10.

FIG 28How do reads of records 2 and 9 affect token 10?

Known: token 9. Goal: select token 10. The first two steps expand two records in one head of one layer; step 3 connects to subsequent model layers.

1Use K to determine how much to read

Q₉Position 9
Its query in this layer

K₂75%

K₉25%

Record 2 comes from the old cache; record 9 is prepared now. Matching Q₉ with K and applying softmax gives assumed retrieval weights of 75% and 25%.

2Weight V coordinates, then add

V₂[2, 0] × 0.75 = [1.5, 0]

V₉[0, 4] × 0.25 = [0, 1]

Add matching coordinates[1.5, 1]

The first coordinate is 1.5 + 0; the second is 0 + 1. This is retrieved information, not yet a token.

3Continue computation, then select token 10

Merge retrieved information intoPosition 9
Finish this layer and pass to subsequent layers

Later layers: still process position 9 to predict position 10

A later reindex layerMay select 5 and 8

A following reuse layerKeeps 5 and 8

Output layer scores the vocabularyToken 10
token

Purple boxes precede the output layer. Some layers reselect records; others reuse the list. Reads can change within the same token-generation step.

Even with the same list, each layer calculates weights using its own Q. Token 10 is selected only after all layers finish. Explanation below

Records 1 and 3–8, unselected here:Their old K/V remains available to later layers. Being unselected does not require recomputation.

Numbers and later-layer choices are illustrative; assume positions 5 and 8 are eligible. 75%/25% allocates reads across history, not token-10 vocabulary probabilities. Actual DeepSeek attention also reads the local window, omitted here. [12][14]

Retrieved information does not immediately become text. Attention output is transformed, merged with position 9's state, and processed by feed-forward and later layers. The output layer finally computes raw candidate vocabulary scores, logits, then argmax or sampling selects token 10. [12]

Different layers may read different history while generating the same token. Earlier layers change position 9's state, from which later layers form their own queries and potentially different scores. Selecting records 5 and 8 later is still part of predicting token 10, not token 11.

V4.1 does not reselect in every layer. Reindex shares historical cache but reselects with the current layer's index query. Reuse also reuses an earlier selection list. The former may change the list; the latter saves another selection operation. [14]

Reusing a list does not reuse its attention weights. Two layers reading records 2 and 9 might allocate 75%/25% and 40%/60%. Each still matches its own main-attention Q against K, allowing different weights and aggregates. These ratios are illustrative. [12]

Noncontiguous records can be read directly. Positions 2 and 9 keep their original indices; they are not renumbered 1 and 2. Reading record 2's already-computed state does not require recomputing the context that produced it. Selection neither deletes nor rewrites history.

The indexer may rescore old index features to choose records. If a later layer selects record 5, it reads the corresponding valid cache. Rescoring and changing the list are not regeneration of old K/V.

Answer 25

Da Wang · explains

Engram is a memory module inside the language model, trained with it. Its learned features reside in a numerical parameter table belonging to the model's weights. Each row, an entry, is a vector. [9][10]

It aims to reduce repeated work recognizing familiar combinations, such as names, places and fixed phrases. The table supplies learned representations before the backbone continues interpreting the sentence and reasoning. [10]

A local sequence of n tokens is an n-gram. Expand its lookup, then compare it with KV caching.

FIG 29Two memories: request state and a learned lookup table

Both paths retrieve data, but its origin differs. The lower panel expands the original “New York” example.

↔ Scroll sideways on small screens; explanations follow each panel.

KV cache: recover state computed for this history

Engram: read learned parameters, then use current context

2The hash chooses the row

This example treats “New York” as two tokens and simplifies lookup to one row. A fixed rule maps the local combination to row 23. Real implementations use several hashes to read multiple entries.

Addresses are known before deep-layer computation. If the table resides in CPU memory, needed rows can be moved to the GPU early, ready when required. This is prefetching. [10]

3Training determines the numbers in the row

These vectors are model parameters, updated against prediction error during training and read by address at inference. Encountering a phrase does not append a new row or retrieve a textual answer. The official demonstration stores entries with nn.Embedding. [9][10]

For a fixed model, table row count and vector width are fixed. Each input position reads only a few rows; the whole table need not be scanned.

4Current context determines how the information is used

Transform the retrieved vector into the backbone's representation, use the current sentence state to decide its contribution, then incorporate it. These are projection, gating and fusion. Computation continues after lookup. [9]

Row numbers, vectors and two-token segmentation are teaching choices, not actual parameters or measurements. The general mechanism follows the official Engram demonstration, whose configuration is not treated as V4.1 production configuration.

“Store every n-gram” needs qualification: the hash table is finite, not a unique collision-free slot for every possible combination. Different combinations may share addresses; multi-head hashing and training address this compression. Unlike exact KV caching's aim to avoid mistaken identities, collisions here are part of the representation's design conditions. [9]

V4.1's official configuration uses 2/3/4-gram memory at layers 2 and 15, zero-based code indices 1 and 14. This is model-specific; the original Engram demo uses different settings. Combining static entries with current textual computation still produces context-dependent results. [12][9]

Answer 26

Da Wang · explains

Distinguish raw data obtained by the program from content actually sent to the next model call. An agent can retain full data, return evidence needed for a decision first and read details later.

FIG 30Which tool-execution data enters the model conversation?

Scenario: read three test logs to locate failures. This example's script arranges storage and filtering. Arrows distinguish external files from model input.

↔ Scroll sideways on small screens; explanations follow each panel.

The model writes a script; Pi executes it and can organize results before returning them

1One script arranges several tool calls

The model writes reading, storage and filtering into a script that Pi executes. This organization is called Codemode.

It can reduce context usage and repeated waits for model replies: a script executes known sequences of tool calls and filters results before returning necessary information. It suits well-defined batch work; new situations requiring interpretation still return to the model.

Illustrative pseudocode:

logs = tools.read_three_logs()
tools.save_full_logs(logs)
text(failure_evidence_and_exit_status)

2Full text stays external and is read when needed

This example stores full logs in files. The model first receives key evidence and file locations, then can read selected sections through tools.

Saving a file or expanding a log in the interface does not mean its full contents entered the model input.

3The outer script remains part of the conversation

The generated script is the outer tool call and usually stays in history. Pi returns the script's explicit output, return value or error; every internal call result need not be returned separately. [32]

Selected results retain failed cases, relevant traces, execution status and full-log locations for follow-up. Pi can also save oversized script output externally and return an excerpt with a reading location. [32]

Answer 27

Da Wang · explains

The change is the standalone tool declarations supplied to subsequent model calls. Pi records the change and, on supported interfaces, expands new descriptions in later messages while keeping the original Codemode directory.

FIG 31Which part changes when a tool is “loaded”?

Continue the log example. The teaching tool log_query is registered in Pi and already callable by Codemode. This round adds it to the tools the model can invoke independently.

↔ Scroll sideways on small screens; explanations follow each panel.

First separate execution capability from declaration: only the bottom row changes

Now inspect input placement: activation does not rewrite the original directory

3Pi predeclares a placeholder to avoid a mid-session miss

This version declares a never-activated placeholder to address the author's reported full cache miss when deferred loading first appears. It is an interface adaptation, not an inherent step in using tools. [20]

For the placeholder's motivation and a fixed-entry alternative, see the follow-up below.

4Configuration organizes the directory; activation does not reorder it

Activating log_query does not move its directory entry or rewrite Codemode instructions merely to reflect that state change.

Writing the new state into the green directory would move the earliest input change before ten rounds of history, affecting all later prefix matching. [20]

5The interface supplies a declaration; Pi executes the call

Supported paths first declare deferred tools with defer_loading: true, then reference or expand them in later messages. Appending ordinary text does not register a tool.

The interface must support mid-session updates. Changed tool definitions or unsupported interfaces cannot guarantee the original prefix remains intact. [31][20]

This diagram separates backend execution, script callability and a standalone model declaration. The API sketch follows the applicable Anthropic path in the pinned Pi version. Directory stability alone does not guarantee every request hits.

Xiao WangWhy not search for log_query only when needed and expand it later? Why predeclare a placeholder?

Da Wang: Specific tools can indeed be discovered when needed. Pi predeclares a never-activated placeholder so the server's accompanying deferred-loading prompt exists from the first round.

Pi's source comments report that the first defer_loading introduces a hidden prompt and that a complete cache miss was measured without the placeholder. Thus, expanding log_query later could still coincide with an earlier server-side change. This is Pi's adaptation to behavior reported by its author, not a universal requirement of dynamic tool loading. [20]

Xiao WangWhy is there a tool prompt? Does structured configuration require enabling deferred loading beforehand?

Da Wang: A tool prompt communicates available tools, argument formats and invocation syntax. Anthropic assembles tool definitions, settings and system instructions into model input as part of its interface implementation. JSON can itself communicate tool definitions; needing that information does not imply first enabling deferred loading must alter the prefix. [34]

Public sources do not provide the complete accompanying prompt or explain why enabling it first would change the beginning. Current interface documentation describes later expansion and cache preservation without listing the first-enable exception in Pi's comment. The evidence explains Pi's motivation, not an inevitable behavior of all current interfaces. [31]

Xiao WangIf designing an agent now, is there a better approach than placeholders?

For log_query, expose only two fixed entries from round 1: search_tools returns names, purposes and schemas; call_tool validates and executes a chosen tool. Keep their names, descriptions and argument formats stable within the session. The following is a teaching interface.

When log access is needed, search through the fixed entry:

search_tools({query: "read test logs"})

The agent returns log_query's purpose and argument information such as run_id in a later message. After reading it, the model invokes the other fixed entry:

call_tool({
  name: "log_query",
  arguments: {run_id: "run_42"}
})

log_query is an argument to call_tool; the model service still sees the same two tool entry points. The agent executes the query and appends its result. Provided entry definitions stay stable, changing business tools need not invoke first-time server-side defer_loading or require its placeholder adaptation.

A practical example is Cloudflare's API MCP service: search discovers API usage and execute runs invocation code. The full catalog stays server-side and relevant portions are retrieved on demand. Its code-execution entry differs from the generic call_tool sketch, but both keep a few stable entry points with operations discovered when needed. [35]

This comparison concerns control over tool changes: fixed entries reduce reliance on platform-native dynamic-tool mechanisms. They still require on-demand usage information, argument validation, permissions and error handling. This does not establish lower cost, faster execution or better correctness on every task than native deferred loading.

Answer 28

Da Wang · explains

Before expected expiry, Pi attempts to replay the original model request with an output limit of one token. This cache keep-alive reuses the input to try to preserve availability. Its response is not added to chat, but its usage is charged to the session. [21]

Introduced in 0.86.0, the feature's defaults are:

Keep alive while the task runs. Stop when the task ends. idle mode additionally tries during inactivity between tasks.
Estimate value first. Usable cache-lifetime information and safely replayable requests are required. Refresh only if estimated net savings reach at least $0.05.
Do not refresh indefinitely. Timing starts from the original real request: at most one hour while running and thirty minutes while idle. If the timer misses the safe refresh window, stop rather than risk rebuilding an expired cache just to keep it alive.

Answer 29

Da Wang · explains

FIG 32Two stages of compaction: write the summary, then continue with it

The agent has read logs and discussed them over several rounds. Replacing older history with a summary requires separate examination of the summary-generation request and the following request.

↔ Scroll sideways on small screens; explanations follow each panel.

Stage 1: does summary generation retain the original prefix?

Stage 2: the summary exists and the next request's history changes

Measure three cost stages separately: summary generation, the first post-compaction request, and later requests. A cache hit while summarizing does not prove the whole old KV remains reusable afterward.

Context compaction changes content sent to the model; KV compression changes how inference state is stored. Replacing full logs with a summary reduces input tokens. Encoding KV numbers in fewer bytes does not automatically reduce API context-token counts.

Answer 30

Da Wang · explains

An API-based agent mainly controls input organization, tool loading, compaction, request scheduling and choice of model/service. Cache location, transport and internal K/V sharing usually belong to the service provider or a self-hosted deployment. See cache placement and reclamation.

Inspect the content actually sent first

Expand and compare adjacent requests' actual tools, system instructions and messages, marking the earliest change. An unchanged chat interface does not imply unchanged input: an agent might rewrite the tool directory or insert a timestamp at the beginning every round.

Keep stable material unchanged when appropriate and append questions/tool results in order. For different questions about one document, separate stable material from the changing question and follow the service's cache rules. Stabilize only content that truly need not change. Updated code or requirements must be reflected; cache hits do not justify stale facts.

Scenario	What to inspect in your agent
Loading tools on demand	Is the opening directory rewritten? Could stable search/execution entries replace it? For native dynamic tools, check later expansion and first-enable behavior.
Large logs or detailed tool output	How large is the raw output, and how much enters the model? Could filtering, external storage and on-demand reads reduce it?
Generating and using summaries	Measure input, cached usage, cost and task outcomes separately for summary generation, the first replacement round and later rounds.
Waiting for long tests	Will the cache remain valid for the next round? Do later savings cover keep-alive costs?

Record usage and diagnose by observed symptoms

Record request purpose, input, cache reads/writes where available, output, cost and duration. Similarly named fields may differ across APIs; use each interface's definitions.

Expand Claude and DeepSeek usage-field details

Claude · Total input has three components

read  = usage.cache_read_input_tokens
write = usage.cache_creation_input_tokens
fresh = usage.input_tokens

total_input = read + write + fresh
hit_ratio = read / total_input if total_input else 0

Claude's input_tokens excludes the cache-read and cache-write components above. Server tools and multi-round aggregates also require their corresponding usage records. [4]

DeepSeek · Hit and miss counts

hit  = usage["prompt_cache_hit_tokens"]
miss = usage["prompt_cache_miss_tokens"]

hit_ratio = hit / (hit + miss) if hit + miss else 0

Field definitions follow DeepSeek's context-caching documentation. [6]

For an aggregate hit rate, sum hit tokens across requests and divide by total input tokens. Do not simply average percentages from requests of different lengths.

Reading old dialogue from cache while writing new dialogue is normal. Separate new-content writes from failed reuse of old content, then diagnose the symptoms below. [4]

Observed symptom	First checks
Old history is not read from cache; the same long content is repeatedly written	Did early input change? Is there a reusable cache point at the end of the stable document? After appending messages, is the previously written boundary still within the service's lookup range?
Only a short opening prefix hits	Where is the earliest change? Were system instructions, tools or project material reordered or rewritten?
Identical content still misses	Does it meet minimum cache length? Has the first write become usable? Has TTL expired? Did the model, API service or workspace change?
High hit rate, but the context window fills quickly	Are obsolete tool results still carried forward? Which evidence must remain, and what can be archived or summarized? Cached tokens still occupy context.
High hit rate, but the task is slow	Time the wait for the first response, generation and tool execution separately. Identify the dominant stage before changing it.

Include every request in whole-task cost

Include the main agent, subagents, summaries, keep-alive calls and retries within one task's accounting. Main-chat cost alone can omit overhead introduced to optimize it. Apply the service's actual billing rules, including applicable cache reads, writes, storage and outputs.

Compare before and after on the same tasks

Use the same tasks and completion criteria. First hold the model fixed and change one strategy at a time. Inspect first use, continuous dialogue and long idle gaps separately, then aggregate whole-task outcomes.

Outcome	Question it answers
Task pass rate	Does the work still finish correctly?
Total task cost	After extra requests, how much money is actually saved?
Start-to-completion time	How much less does the user wait? Report typical and slower tasks separately.
Cache usage and stage timings	Where did gains or losses occur?

You can also compare total cost of the task set ÷ accepted task completions, while reporting pass rate alongside it. Mark unavailable cache-size or internal-restoration measurements as not directly measurable; do not present estimates as measurements.

When changing models or services, repeat the same-task comparison of completion quality, cost and time.

Answer 31

Da Wang · explains

Use a cheaper method to propose candidates, then verify them with the target model responsible for the answer. This generation method is speculative decoding. Here, a smaller draft model assists the target model.

FIG 33Generate the same A, B and X while waiting for fewer target-model passes

Continue from the same committed prefix, choosing the target model's highest-scoring token at each step. A/B/C/D are draft candidates. The example accepts A/B and replaces C with X. Each letter is one token.

① Ordinary generation: three sequential target passes

Committed prefix

Target model · pass 1Select A

Target model · pass 2Select B

Target model · pass 3Select X

Committed new contentABX

② Speculation: cheaper draft + one target verification pass

The same committed prefix

Draft model · sequential unverified proposals

A→B→C→D

Send the candidate sequence to the target

Target model · one batched pass

A ✓B ✓Replace C with XDiscard D

Committed new contentABX

Acceleration is conditional:The draft must be cheap enough and enough candidates accepted to outweigh added work. Frequent rejection can make speculation slower.

This compares execution steps; box width is not elapsed time. Three passes becoming one does not imply 3× speedup. Candidates and verification outcomes are illustrative. The next question expands the batched computation. [54][60]

Answer 32

Da Wang · explains

This chapter uses two independent models with different parameters. The draft supplies candidate token IDs. The target looks up its own embeddings and projects its own Q/K/V. Randomized verification additionally needs draft probabilities for acceptance and resampling; those are not draft K/V reused as the target's cache.

This example retains full historical K/V by position. Fix the starting state: tokens 1–9 are committed; the target cache reaches position 8; token 9 was just selected and has not been fed back. The draft proposes A/B/C/D at positions 10–13.

FIG 34Transmit candidate content; the target recomputes its own state

Purple denotes draft state; green denotes target state. Solid arrows show data transferred this round.

The pinned implementation obtains candidate tokens and available draft scores, then calls the target to process candidates and obtain its own predictions. [61]

Ordinary generation waits for the target's next token to know the next input. Here, the draft already supplies A, B and so on. The target computes the predictions conditional on that proposed sequence. If an earlier candidate is rejected, later results depending on it cannot join the committed history.

Answer 33

Da Wang · explains

Participating in model computation and producing prediction scores are different. Final hidden vectors are not vocabulary scores. Passing selected positions through the output layer produces raw candidate-token scores, or logits.

FIG 35The same row of hidden vectors, but different positions reach the output layer

Colored cells represent hidden vectors; small bars represent vocabulary scores. Column numbers are token positions.

Parallel ① Dashed boxes with matching numbers can run concurrently; arrows still show dependencies. Numbers match only within this diagram.

① Ordinary input: five input tokens, then the first response token

12345

Final-layer internal vectors

↓

Output projection produces vocabulary scores

Not selected

Predict 6Begin response

② Speculative verification: predictions around several candidate positions

910 · A11 · B12 · C13 · D

Final-layer internal vectors

↓↓↓↓↓

Output layer: score these positions as a batch

Parallel ①

Output layerScores

Parallel ①

Output layerScores

Parallel ①

Output layerScores

Parallel ①

Output layerScores

Parallel ①

Output layerScores

Vocabulary scores for each position

Predict 10Verify A

Predict 11Verify B

Predict 12Verify C

Predict 13Verify D

Predict 14If all accepted
Extra token

Four draft candidates require four verification predictions, usually plus one bonus-token prediction. Other old cached positions need not all recompute their vocabulary scores. [61]

Implementations supporting the option set logits_to_keep = candidate_length + 1. Batched input does not imply retaining only the final prediction. The output layer processes more positions, and ultimately discarded candidates may already have incurred computation.

Vocabulary scores select the next token. Attention matching scores control a layer's historical reads. They are different scores.

Answer 34

Da Wang · explains

Remove the draft first: how would the target choose?

Fix the preceding context and select one immediately following token. The target scores vocabulary candidates, then the sampling settings determine their selection probabilities.

Greedy decoding always selects the highest-probability candidate, as in the A/B/C→X example. Random sampling selects according to probabilities: likely candidates appear more often, but others remain possible. This question concerns random sampling.

What exactly does “20%” describe?

For illustration, this position has three candidate tokens: A, B and C. They are alternatives at one position, not three consecutive tokens. Probabilities below are teaching choices.

Candidate token	Target selection probability
A	20%
B	50%
C	30%

Greedy decoding selects B, whereas random sampling may also select A or C. 20% means A's chance at this immediately following position, holding the prefix, model and sampling settings fixed.

It is not a 20% chance that A is correct, nor a requirement for twenty A tokens in a hundred-token response. Once a token is selected and generation continues, context changes and the next position gets a new distribution.

Adding a draft can change the probabilities

A cheaper draft proposes candidates before batched target verification. Because its parameters differ, it can assign different probabilities even for the same prefix:

Candidate token	Original target probability	Draft proposal probability
A	20%	60%
B	50%	30%
C	30%	10%

Accepting every proposal makes the final choice follow the draft column. A would rise from the target's 20% to 60%. Drafting would then change the generation rule.

What should verification preserve?

Use the draft's advance proposals while ultimately selecting according to the target's 20%, 50%, 30%. Three steps are needed:

The draft samples a candidate from its own distribution.
Using both models' probabilities, the verifier accepts or rejects it.
On rejection, resample from a specifically corrected distribution.

Correction changes acceptance and resampling rules, not either model's trained parameters. Work through the same numbers to see which proposal mass survives and what replaces rejected mass. [54]

FIG 36Draft A: 60%. Target A: 20%. How do we correct it?

Target: A 20%, B 50%, C 30%. Every full bar represents 100% probability, not a token count.

① Accept: retain one third of A proposals, all B/C proposals

CandidatesSolid: directly accepted · Red: rejectedDraft

A

20%40%

60%

B

30%

30%

C

10%

Directly accepted 60%To resampling 40%

② Resample: split the rejected 40% equally between B and C

Resample after rejection40%

B · 50% on resampling＋20%

C · 50% on resampling＋20%

③ Combine: recover the target's 20% / 50% / 30%

CandidatesSolid: directly accepted · Hatched: resampled contributionFinal

A

20%

20%

B

30%＋20

50%

C

＋20

30%

A: 20 + 0B: 30 + 20C: 10 + 20

These are worked probability calculations, not a model measurement. One draw selects one token at this position; the diagram shows each outcome's chance. Acceptance and resampling rules were checked against the paper and pinned implementation. [54][60]

From distribution mismatch to average one-step acceptance

Keep the same prefix and next-token position. Let p be the target distribution, d the draft distribution and i range over vocabulary candidates. Both distributions must already include temperature, truncation and other actual sampling settings.

Sum the absolute differences in candidate probabilities and divide by two to obtain total variation distance, TV: the probability mass not shared by the distributions.

TV (p, d) = \frac{1}{2} \sum_{i} | p_{i} - d_{i} |

Under standard speculative sampling, before a candidate is drawn, average one-step acceptance α is:

α = \sum_{i} \min (p_{i}, d_{i}) = 1 - TV (p, d)

Candidate i is proposed with probability d_i and accepted conditionally with min(1, p_i/d_i). Their product is min(p_i, d_i). Summing these contributions gives α. [54]

Substitute the three candidates above

p＝(0.20, 0.50, 0.30)，d＝(0.60, 0.30, 0.10)。

TV = (0.40 + 0.20 + 0.20) / 2 = 0.40; α = 1 − 0.40 = 60%, matching the diagram's directly accepted mass.

That does not mean every candidate has a 60% conditional acceptance rate. Once A is proposed, its chance remains 1/3. Average acceptance is also not actual speedup.

Paper §3.2 denotes this TV quantity by D_LK. It differs from KL divergence's D_KL discussed earlier; similar notation does not make them interchangeable. [54]

Each new position may change the prefix and both distributions. Verification retains only the consecutively accepted beginning. One position's TV is not a fixed acceptance rate for an entire draft. Consider accepted-prefix length together with execution costs.

Xiao WangIf target A is 60% and draft A 20%, their ratio is 3. Does that exceed 100%, and is A always accepted?

Da Wang: Yes: conditional on the draft having proposed A, accept it with certainty. Acceptance is capped at one, or 100%. There is no 300% acceptance probability.

Always divide target probability by draft probability, then cap at one: acceptance = min(1, target probability / draft probability). Here 60% / 20% = 3, giving 100% acceptance.

Event	Probability here
The draft proposes A	20%
Accept A, conditional on its proposal	100%
Select A via proposal followed by direct acceptance	20% × 100% = 20%

Always accepting a proposed A does not make every final token A. The draft proposes it only 20% of the time, so this path contributes 20% overall. The target needs 60%; the missing 40 percentage points come through resampling after other candidates are rejected. [54]

The earlier diagram fixes the opposite numbers: draft A 60%, target A 20%, giving 20% / 60% = 1/3 acceptance. The supplied probabilities changed; the direction of division did not.

Answer 35

Da Wang · explains

FIG 37X is selected but does not yet have its own K/V

Observe immediately after verification: accept A/B, replace C with X. Horizontal numbers are token positions; aligned rows compare text and target K/V.

① After verification: committed text reaches 12, usable K/V reaches 11

Position1～910111213

Committed tokens Through position 12

Existing content

A

B

1X

Undecided

Currently reusable K/V Through position 11

KV

1～9

KV

A

KV

B

Not yetX's K/V

—

Computed this round, removed from active history

2

KV

C

KV

D

② Next round: feed X into the target to create its K/V

Input at position 12X

3Target modelReuse K/V at positions 1–11

KV

X's K/VAppend position 12

Scores predicting position 13Select the next token from these scores

Crossed-out records stop being read; whole memory pages are not necessarily cleared immediately. A page containing retained records or shared with another request cannot be reclaimed wholesale. See paging and copy-on-write. [30][47]

Prefix caching and speculative decoding can work together: the former reuses committed old input state; the latter tries to commit new content faster.

Xiao Wang Four draft steps were computed, but only two accepted. With explicit K/V, the final two records can be excluded. If all four updated the same state, how do we restore the moment after the first two, and what must be retained beforehand?

Da Wang: Still use A/B/C/D. Restore the per-layer state after processing B, then feed X. Fixed-size state has no four tail records to truncate. Choose a method from what was retained:

A snapshot after B: restore every layer's state from that moment.
Only a snapshot before A: restore it and replay A/B along the original computation.
Reversible updates with sufficient retained contributions: for the additive example, subtract C/D's per-layer matrix and vector contributions in real arithmetic. Floating-point subtraction need not recover the original bits, and other recurrent updates need not support this operation.

A snapshot must protect the old numerical values. Keeping a reference to an array later mutated in place does not preserve old state. Copy the data or use storage that protects the previous values. [69]

Answer 36

Da Wang · explains

Arithmetic work and waiting time need not change together. Batched verification reuses loaded parameters across positions and may exploit otherwise idle computation. Drafting, additional prediction positions and rejected computation add overhead.

What do KL, TV and actual speedup each tell us?

Draft–target KL divergence alone cannot establish actual speculative-decoding speedup. Let p denote target and d draft probabilities under the same prefix and actual sampling settings.

Quantity	What it measures	How to interpret it
KL divergence	A directional distribution discrepancy: the reference-weighted average log probability ratio.	Useful for distribution matching, but not one-step acceptance and not a measure of runtime.
TV	Probability mass not shared by the distributions.	Under the standard sampling rule above, 1−TV is average one-step acceptance for the same prefix. It does not directly specify consecutively accepted draft length.
Actual speedup	Ordinary-generation time divided by speculative-generation time under comparable tasks, hardware, concurrency, sampling and other conditions.	Also account for accepted-prefix length, committed tokens per round, and drafting, target verification and state-management costs.

Execution costs include draft generation, actual batched target runtime, parallel efficiency, temporary candidate K/V and rollback, transfers, communication and scheduling waits. Additional memory may reduce concurrent capacity. None of these costs appears in KL or TV. [54]

Even identical distributions at every step, KL = TV = 0 and complete draft acceptance do not guarantee a speedup. If drafting already takes nearly as long as ordinary generation, verification and state management can make the total slower. Compare actual progress per round with its time below.

FIG 38How can computing more positions reduce waiting?

Parallel ① Dashed boxes with matching numbers can run concurrently; arrows still show dependencies. Numbers match only within this diagram.

① One loaded weight tile serves linear computations at several positions

② An illustrative time ledger: both commit A, B and X

0306090 ms

Ordinary generation90 ms

A · 30B · 30X · 30

Speculative decoding44 ms

Batched verification · 40

Draft: 4 msThe round also commits A, B and X

Times are teaching assumptions, not measurements. Bars follow those assumed proportions. Actual benefit depends on acceptance, draft cost, hardware and concurrency. [54]

Xiao Wang A/B/C/D are available, so the target can process them as one input batch within every layer. Loaded weights can serve several positions rather than repeatedly reading data for each next generation step.

Can that reduce transfers between GPU memory and on-chip storage? FlashAttention also tiles computation and reuses loaded data on-chip. Do they avoid the same overhead? Is speculative decoding an upgraded FlashAttention?

Da Wang: Speculation can reduce some repeated reads, but the techniques optimize different stages; it is not a FlashAttention upgrade. Here we compare transfers between GPU memory and fast on-chip storage.

Separate learned model parameters W from Q/K/V, numerical intermediates computed for the current input.

Comparison	Speculative decoding	FlashAttention
Initial bottleneck	Future token identities are unknown, usually requiring selection before the next token is processed.	Given Q/K/V, attention produces large intermediate scores and weights that move between GPU memory and the chip.
How transfers are reduced	Draft candidates let the target process several positions in one pass, potentially sharing loaded parameters and historical K/V across them and avoiding repeated per-round reads. [54]	Tile Q/K/V and consume scores in subsequent on-chip attention computation, avoiding full score/weight tables written to and read from GPU memory. [49]
Scope	Generation flow: draft, verify, accept and roll back.	Attention computation inside model layers.
Can they be combined?	Yes. The target's attention can use FlashAttention during batched speculative verification.

Answer 37

Save a public UTF-8 document as public_notes.txt, long enough for the chosen model's minimum cache requirement. Keep model and other settings fixed across the four requests and follow the table's order. Establish the cache before comparing changes. [4]

Request	The only change	Observation
1 · Prepare cache	Submit this experiment's document and ask for a one-sentence summary.	Does cache-write usage appear?
2 · Confirm hit	Repeat request 1's entire input.	Does cache-read usage appear? Stop without a confirmed hit.
3 · Change question	Keep the document; replace the final question with “List three key points.”	Is the document's cached prefix still read?
4 · Change the beginning	Keep question 3; change only the document's opening version marker from A to B.	How do reads and writes change after the early modification?

A unique run identifier is shared by the four requests, reducing accidental reuse of earlier experiments. The cache marker ends the document; the question follows in a user message. Prior answers are not carried forward, keeping comparisons clear.

Running this costs API credits. At most four requests are sent, each limited to 96 output tokens. Errors or an unconfirmed second-request hit stop execution without automatic retries. Code is provided without invented measurement results.

Expand the complete Python probe and instructions

Use Python 3's standard library only. Save the code as kv_cache_probe.py and replace YOUR_MODEL_ID with an available prompt-caching model ID. The API key comes from an environment variable or a non-echoing prompt.

python3 kv_cache_probe.py public_notes.txt --model YOUR_MODEL_ID > kv-probe.jsonl

#!/usr/bin/env python3
"""Four sequential Claude API requests; running this file incurs API charges."""
import argparse
import getpass
import json
import os
from pathlib import Path
import sys
import time
import urllib.error
import urllib.request
import uuid


def request_once(case, prefix, question, model, api_key):
    payload = {
        "model": model,
        "max_tokens": 96,
        "stream": True,
        "system": [{"type": "text", "text": prefix,
                    "cache_control": {"type": "ephemeral"}}],
        "messages": [{"role": "user", "content": question}],
    }
    request = urllib.request.Request(
        "https://api.anthropic.com/v1/messages",
        data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
        headers={"content-type": "application/json", "x-api-key": api_key,
                 "anthropic-version": "2023-06-01"},
        method="POST",
    )
    started = time.perf_counter()
    first_text = None
    usage = {}
    complete = False
    error = None
    response_model = None
    stop_reason = None
    try:
        with urllib.request.urlopen(request, timeout=180) as response:
            for line in response:
                if not line.startswith(b"data:"):
                    continue
                event = json.loads(line[5:].strip())
                kind = event.get("type")
                if kind == "message_start":
                    message = event["message"]
                    usage.update(message.get("usage", {}))
                    response_model = message.get("model")
                elif kind == "message_delta":
                    # Output usage is cumulative: replace it, do not add it again.
                    usage.update(event.get("usage", {}))
                    stop_reason = event.get("delta", {}).get("stop_reason")
                elif kind == "content_block_delta":
                    delta = event.get("delta", {})
                    if delta.get("type") == "text_delta" and delta.get("text"):
                        if first_text is None:
                            first_text = time.perf_counter() - started
                elif kind == "error":
                    error = event.get("error", {}).get("type", "stream_error")
                    break
                elif kind == "message_stop":
                    complete = True
                    break
    except urllib.error.HTTPError as exc:
        error = f"HTTP {exc.code}"
    except (OSError, ValueError) as exc:
        error = type(exc).__name__
    if not complete and error is None:
        error = "stream_ended_without_message_stop"
    inputs = [usage.get(k) for k in (
        "input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens")]
    return {
        "case": case, "requested_model": model, "response_model": response_model,
        "first_text_s": round(first_text, 3) if first_text is not None else None,
        "request_s": round(time.perf_counter() - started, 3),
        "total_input_tokens": sum(inputs) if all(type(v) is int for v in inputs) else None,
        "usage": usage, "stop_reason": stop_reason,
        "complete": complete, "error": error,
    }


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("document", help="UTF-8 public reference text")
    parser.add_argument("--model", required=True, help="An available Claude model ID")
    args = parser.parse_args()
    document = Path(args.document).read_text(encoding="utf-8")
    if not document.strip():
        parser.error("The reference text is empty.")
    api_key = os.environ.get("ANTHROPIC_API_KEY") or getpass.getpass("API Key: ")
    if not api_key.strip():
        parser.error("An API key is required.")
    run_id = uuid.uuid4().hex
    prefix = f"Experiment ID: {run_id}\nDocument version: A\nAnswer using the following reference.\n{document}"
    changed = prefix.replace("Document version: A", "Document version: B", 1)
    cases = [
        ("1_prepare", prefix, "Summarize this reference in one sentence."),
        ("2_confirm", prefix, "Summarize this reference in one sentence."),
        ("3_new_question", prefix, "List three key points from this reference."),
        ("4_changed_prefix", changed, "List three key points from this reference."),
    ]
    for case, material, question in cases:
        result = request_once(case, material, question, args.model, api_key)
        print(json.dumps(result, ensure_ascii=False), flush=True)
        if not result["complete"]:
            print("Incomplete request; stopped. Known usage is retained; no automatic retry.", file=sys.stderr)
            return 1
        if case == "2_confirm" and not (result["usage"].get("cache_read_input_tokens") or 0):
            print("Cache hit not confirmed; stopped. Check document length, caching rules and TTL.", file=sys.stderr)
            return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Each result-file line describes one request. usage retains returned API usage; first_text_s measures request start to the first visible text fragment, and request_s the complete duration. Missing fields remain unknown, not zero. Incomplete requests retain usage received so far. Interrupted records may be incomplete; the service bill is authoritative.

Calculate charges using the selected model's actual ordinary-input, cache-write, cache-read and output prices, recording their date. Inspect requests separately before totaling the entire experiment, including initial cache creation.

If request 2 misses, check length, marker and TTL first. A hit on request 3 followed by reduced reads or renewed writes on request 4 supports prefix-dependent reuse in this run. One run does not establish stable speedup. This tests caching behavior; whole-task benefits require the earlier evaluation method.

Answer 38

The experiment uses a three-layer causal Transformer: four Q heads, two KV heads, eight dimensions per head, RoPE, normalization, feed-forward and output layers. A fixed random seed initializes weights; inputs are integer vocabulary IDs.

FIG 39Turn expected equivalence into three executable controls

The minimal model initializes from scratch without model downloads. It exposes computation dependencies and cache-management mistakes.

↔ Scroll sideways; numbered notes follow their corresponding panels.

Fix weights, positions and inputs across three paths

42/45/47/32/15 are actual input constructions and recorded counts from the appendix program. Complete scripts, automated tests and raw JSON accompany Appendix C. [59]

PrefixCache stores sixteen-token blocks with weight identity, namespace and a chained prefix identity. Hits copy into private state. A separate PagePool simulator tests shared reference counts and copy-on-write. Keeping them separate avoids presenting a page-management class as an implemented GPU paging kernel.

Accompanying run results	Hit	Computed	Cached vs full: maximum absolute difference	Cached vs split: maximum absolute difference
float64	32	15	0	0
float32	32	15	0	0

Both recorded precisions show zero difference in these comparisons in the supplied environment. Reruns should retain their own values and test tolerances; different hardware and matrix shapes may produce small differences.

Other controls	Supplied result	What it establishes
Different equal-length context: maximum first-layer document-K difference	0	This model's first layer transforms positions independently before reading history
Maximum second-layer K difference for the same document	2.112007	Higher-layer state already depends on different context
Maximum next-token logit difference after an invalid splice	1.311954	Splicing can be wrong even with unchanged positions
Maximum difference after replaying the document under the new prefix	0	Full replay is the correct-repair control, not a selective-repair result
Maximum online-softmax vs full-result difference	5.55e-17	Checks global accumulator merging
INT4 payload + scales for 2,048 numbers	1,024 + 512 = 1,536 bytes	Actual array bytes, excluding Python objects and runtime workspace

Xiao WangShould I read all the code before running it?

Da Wang: Predict hit length and which layer an invalid splice affects, then run tests. Read TinyDecoder.forward → PrefixCache.lookup → store, followed by online_attention and PagePool. Inspect one mechanism at a time rather than every implementation detail together.

Provider	Model checked	Adoption and distinctions
DeepSeek	DeepSeek-V4.1-Flash	Explicitly inspired by YOCO, with modifications. The report states this influence and combines it with grouped sharing; public code reuses K/V produced by other layers. [14][12]
GLM / Zhipu	GLM-5.3 GLM-5.3-Flash	The inspected implementation is not YOCO. GLM-5.3's IndexShare shares selected historical positions, while K/V remains per-layer. Flash mixes linear and sparse attention, with state maintained separately by each layer. [15]
Kimi / Moonshot AI	Kimi-K3	The inspected implementation is not YOCO. It uses hybrid attention and distinct per-layer state; historical-K/V layers independently generate their caches. [16]
Qwen / Alibaba	Qwen3.8-2.4T-A95B Qwen3.8-Flash-Next	The inspected implementation is not YOCO. It uses hybrid attention; historical-K/V layers produce their own K/V rather than having several later layers read one earlier-half cache. [17]
Google	Gemma 3n	Related cross-layer sharing is implemented. Intermediate layers produce local and global K/V for several subsequent layers to share. [18]

Row	Start	Duration	Total input	write	read
1	00:00	20s	8,500	8,500	0
2	00:30	15s	10,000	1,500	8,500
3	00:55	18s	11,500	1,500	10,000
4	01:20	25s	32,800	21,300	11,500
5	01:55	15s	34,650	34,650	0
6	02:20	150s	36,150	1,500	34,650
7	07:55	20s	37,650	37,650	0
8	08:25	15s	39,150	1,500	37,650
9	08:50	15s	40,650	33,300	7,350
10	09:15	15s	42,150	3,000	39,150
11	09:40	20s	51,870	44,520	7,350
12	10:10	25s	82,370	30,500	51,870
13	10:10	25s	82,370	30,500	51,870
14	10:10	25s	82,370	30,500	51,870

Row	read	What can be established	What still needs checking
5	0	Input grows by 1,850 and starts just 35 seconds later. Natural five-minute expiry alone does not explain the miss. Compared with the later 7,350, the extra 350 is a priority lead.	A total-volume change cannot locate the addition. Inspect tools, system, messages and configuration.
7	0	335 seconds since row 6 started, but 185 since it finished. With start-based TTL and no other refresh, expiry is sufficient to explain the miss.	Use request start time; do not generalize to all real-service cache entries expiring together.
9	7,350	With minimum full input 8,500 and only two write boundaries, the hit is tools+system, 350 above its initial 7,000.	Whether the 350 entered tools or system, when it entered and why messages missed remain unknown.
10	39,150	The full prefix from row 8 matches again, showing that older entry remains available.	A temporary configuration change restored afterward or a temporary history rewrite restored afterward could both explain it; numbers alone cannot distinguish them.
11	7,350	The hit again stops at tools+system; the next three requests can read this row's full prefix.	Check history/configuration changes and whether new content blocks pushed the old entry outside the lookback range.

Row	Change relative to the parent
1	Initial request: tools + system + user task, 1,500 tokens
2、3、6–10	Each adds a 1,500-token tool round: 300 model output tokens, text plus one tool_use, and a 1,200-token tool result
4	Add a round whose tool result is a 21,000-token log
5	Add a round; the agent wrapper inserts db_query, 350 tokens, into the middle of tools
6	150 seconds from request start to response completion
7	A local test runs for 3 min 05 sec before sending
9	Change tool_choice from auto to forced submit_patch
10	Restore tool_choice to auto
11	Append 120 tokens of final model reply; user adds 24 files, one content block each, totaling 9,600
12–14	Three subagents send in the same second. Each input = all row-11 input + the same 300-token model reply + the same 30,000-token repo map + its own 200-token instructions. Breakpoints are unchanged.

Row	Actual cause	Fix	Revised write / read	Input-equivalent saving when changed alone
5	Inserting a tool definition breaks the original prefix.	Stabilize the deferred-loading mechanism from round 1; add tools through supported discovery/reference paths later. Details follow the table.	1,850 / 32,800	36,915
7	Row 6 starts at 02:20 and expires at 07:20; the next request starts at 07:55.	Safely replay at 06:50 with unchanged cache identity, reading 36,150; bill using five-minute TTL.	1,500 / 36,150	37,957.5, after keep-alive cost
9、10	Row 9 changes tool_choice; row 10 restores it and reconnects an older prefix.	Removing forced invocation depends on the required guarantee; see below.	1,500 / 39,150；1,500 / 40,650	38,295, if the model complies
11	25 new positions push the previous endpoint to lookback position 26, beyond the limit of 20.	Combine files into one block, or explicitly mark the preceding request's final block in the history carried by the current request.	9,720 / 42,150	40,020
12～14	Requests start together and lack a breakpoint at the common repo-map endpoint.	Mark the common endpoint; send A first, then B/C after its entry becomes readable.	A unchanged; B/C each 200 / 82,170	69,690

Item	Original	All fixes
Requests	14	15, including one extra keep-alive
Written tokens	280,420	82,770
Read tokens	311,760	545,560
Token hit rate	52.65%	86.83%
Input cost: ordinary-input-token equivalents	381,701	158,018.5

Path	Ordinary-input-token equivalents
Always force	50,025
Auto first; model complies	11,730
Auto first; noncompliance, then forced retry	54,090

Computational paths compared	fp32	bf16
① Repeat the same algorithm twice	Bitwise identical	Bitwise identical
② Cached versus uncached, processing all 1,230 tokens at once	3.7e-5	0.33
③ Without cache-table lookup, process the second request as 1,200 then 30 tokens versus all at once	3.7e-5	0.33
④ Cached versus the split computation in ③	Bitwise identical	0.37
⑤ K/V for the first 1,200 tokens computed in request 1 versus request 2	Bitwise identical	Median relative error: 0.6%
⑥ First layer's first matrix product for the last 30 tokens: within 1,230 rows versus a standalone 30-row computation	9.5e-6	Not measured
⑦ Full-sequence computation alone versus batched with another request	3.8e-6	Bitwise identical

Observation	fp32	bf16
Per-step ε: median / maximum	1.4e-5 / 5.4e-5	0.26 / 1.0
Steps with m ≤ 2ε, using greedy argmax	0	95
Steps selecting different tokens, greedy	0	8
Same, temperature-1 sampling with common random numbers	0	9

Method	KL	Most likely token
Valid prefix cache, experiment 1 row ②	5.8e-8	Same
Append after different content of the same length	3.1e-2	Same
Append after a prefix of different length	6.0e-2	Changed

Scenario	Method	Reported result
B arrives halfway through A's decode; insufficient free pages	No ownership count, only recency order	B overwrites A's page; 12 of A's next 16 tokens differ, maximum logit difference 24.5
Same scenario	Count active request users	Reject B until A finishes; A remains bitwise identical
A1/A2 share the first 11 pages; A1 finishes, then B arrives	Only an “in use” boolean	B overwrites a shared page; 19 of A2's next 20 tokens remain unchanged, maximum logit difference 5.2
Same scenario	Count active request users	Bitwise identical
A returns after one page is evicted	Evict the chain's earliest page first	Hit 0/195; the remaining 11 pages become unreachable. Recomputation is bitwise identical to uncached execution.
Same scenario	Evict the chain's final page first	Hit 176/195; maximum recomputation difference 6.3e-5, all 12 token choices identical

Observation	bf16 greedy	Temperature 1, common random numbers
Prompts with a first difference within 150 steps	3 of 5	4 of 5
First-difference positions, sorted	22、32、54	8、11、20、79
No difference observed by the endpoint	2 prompts	1 prompt
All difference counts in prompt order	1, 0, 4, 3, 0; total 8	2, 2, 2, 3, 0; total 9

Design	When protection may end
Copy into private request-owned data	After copying completes and the request no longer depends on the source.
Read shared pages at every step	After the final user stops reading those pages.

Scheme	Retained state	What each layer reads next
A · Full historical K/V	Each of 32 layers keeps full K/V. Per token, K and V each have two heads of 64 numbers.	Each layer reads its own 4,096-token K/V history once.
B · Shared global + local windows	Four groups of eight layers each share a global K/V history shaped as in A. Every layer additionally has an independent 128-token local window with the same K/V shape.	Every layer reads its group's entire global history and its own 128-token window. The two states are computed separately, not merged into one read.
C · Fixed-size recurrent state	Every layer retains a 64×256 matrix, updating it from the current matrix and this layer's input for each processed token.	Each layer reads its whole matrix once and produces its updated matrix, without per-token historical K/V reads.

Scheme	Stored size	Sessions within 640 MiB
A	2 MiB per layer × 32 = 64 MiB	10 same-length sessions
B	Four global histories: 8 MiB; 32 local windows: 2 MiB; total 10 MiB	64 sessions
C	32 KiB per matrix × 32 layers = 1 MiB	640 sessions

Scheme	Calculation	Logical read volume
A	32 layers each read 2 MiB of history	64 MiB/step
B	32 layers each read 2 MiB shared global state plus 64 KiB local state	66 MiB/step
C	32 layers each read a 32 KiB current matrix	1 MiB/step

Scheme	Where continuation is currently possible	Snapshot that would change the result
A	Every layer retains K/V for document positions 1–3,072. Take that prefix and exclude the old-question suffix to process the new question.	No additional intermediate snapshot is missing; full history is already retained.
B	Global document K/V remains, but each layer's window holds only old-question positions 3,969–4,096. Document-end local state is missing and must be reconstructed.	Snapshot all 32 local windows after token 3,072: 2 MiB. Together with retained global prefix state, this restores the endpoint.
C	Only matrices after token 4,096 remain; no permitted rollback is available. Sequentially replay the document's 3,072 tokens from initial matrices.	Snapshot all 32 matrices after token 3,072: 1 MiB. Load them and process the new question.

Common-prefix endpoint marker	Send timing	Question being distinguished
Absent	Simultaneous	Neither a dedicated entry nor a wait: baseline control.
Absent	Send B after A begins responding and provides a write receipt	Can waiting alone reuse a common prefix that was never separately retained?
Present	Simultaneous	The endpoint is written this round, but is it readable when B begins processing?
Present	Send B after A begins responding and provides a write receipt	After waiting for documented readability, can the full common prefix be reused?

Entry point	What to trace
TinyDecoder.forward	Positions include cached length; RoPE transforms current Q/K; append KV heads; attention cannot read future positions
PrefixCache.lookup / store	Chained full-block identity; stop at first miss; private copies on hits; capacity and eviction
online_attention	Per-tile maximum, denominator and weighted accumulator; one final global normalization
pack_int4 / unpack_int4	Actually pack two codes per byte; scales consume separate space; this is not NVFP4
PagePool	Reference counts, release and copy-before-branch-write; this simulator does not drive model attention

Which layers of the system determine the practical benefits of KV caching?

First distinguish model layers, attention heads and token positions

Model layer 1

Attention: 4 query heads

Model layer 2

Attention: this layer's own 4 query heads

How do these experts relate to attention heads? If two experts are selected, do we get two additional sets of historical K/V?

Attention within this layer

The MoE feed-forward component

How does MoE affect caching benefits?

Seven groups of factors jointly affect cache use

Agent architecture

Input and model configuration

Model architecture

Representation and compression

Inference framework

Cache storage and transfer

Hardware and workload

Observe eight outcomes separately

Stored size and peak memory

Cache hits on repeated input

Restore time

Time to first token

Generation speed

Throughput under concurrent use

Total cost

Answer and task correctness

Why does repeated input deserve separate attention? How do the computations avoided by caching affect a whole task's cost and duration?

One input → one answer

Return history → compute repeatedly → verify

What are K and V, what does each do, and exactly which computations can the next step avoid by retaining them?

How does inserting image vectors into the input sequence differ from preparing separate image K/V? Does the text query have more records to read?

①No image: 5 text positions

②Image in the sequence: 4 image + 5 text positions = 9

③Separate image memory: 5 text positions plus a distinct image read

If both input and output compute Q/K/V, why can input be batched while output is usually more expensive?

① Known input: each position has its own starting point

② Attention in this layer: restrict visibility without queuing rows

③ Next layer: previous context is already in each hidden state

Why can't future response tokens also be processed in advance?

Process input first: all three tokens are given

Continue generation: D is this step's new input

1Input: process several known positions together

2Generation: the current token still permits parallel work

3Why can Q/K/V depend on history without directly reading it?

If the current token and its position match, why can't we directly reuse the previous K/V for “apples”? How does earlier context affect it?

Can this correctly distinguish contexts? Does “include the entire prefix” mean storing its plaintext or storing an identifier that distinguishes it?

The same “apples” can have different prefix IDs

Why not make lookup fine-grained enough to reuse 100% of the matching prefix? What does grouped management save that justifies the extra computation?

Creation cost: first process the same million-token document

Reuse benefit: two divergence points in a 1,000-token request

What does this marker actually control? Does it cache only that block or everything from the request's start through that point? Is the new question after it still processed?

At the marker, distinguish model computation from service-level caching

How does it discover and store the common opening? Can the second request reuse the document, or might reuse begin only later?

Distinguish global K/V from local K/V first

Global K/V: older content remains readable

Local K/V: recent state retained separately by each layer

If A was computed already, why might C still be unable to continue from it?

Follow the numbers: retain, evict, restore

How does V4.1 change this constraint?

When the user returns, how does the GPU find and correctly read a cache moved elsewhere or scattered across GPU memory?

1Idle state can leave GPU memory

2Once in GPU memory, locate records through a table

3Separate a writable tail page before two answers diverge

Why do those temporary results consume so much GPU memory, and how does FlashAttention address it?

The problem: two large temporary tables beyond K/V

FlashAttention: finish a tile's consecutive operations on-chip

Separated implementation: write a table to GPU memory, then read it back

FlashAttention: load a tile and continue its computation on-chip

What if exponentials become too large?

What is eliminated, and what remains?

How should cache capacity be calculated? What do GQA and MQA store less of?

Query heads 1–4

Query heads 5–8

Who shares K/V in GQA/MQA versus YOCO?

How does KV quantization save actual bytes, and why can smaller storage run more slowly?

The same 16 numbers

The reconstruction path also affects speed

Reconstruct the whole range first

Reconstruct while reading