How does one token’s
cache become
a shared memory?

Picture an assistant that reads code, runs a terminal and fixes errors. The first time, you give it the project files; the second time, it comes back with test results; the third time, it keeps editing. The same tool descriptions, project conventions and conversation history show up in request after request.

So a useful question to ask of Flash-class models is: at the same acceptance standard, how much did the whole task cost, how long did it take, and how many retries did its failures need? Cheaper input processing can change that answer.

In plain words first: after a model reads a position, it leaves behind information that later positions can query. In attention, K (Key) helps decide “which position to attend to,” V (Value) supplies the content that gets read out, and Q (Query) stands for the current lookup. All three are vectors of numbers: not the original text, not a summary of an answer, and not an ordinary database record.[1]

A single call usually does two kinds of work. Prefill processes a long input that is already given, and can usually compute many positions in parallel batches. Decode, in ordinary autoregressive generation, produces the next token from what came before. The cache lets later computation use the K/V already produced, instead of encoding the old input again and again.[1]

“Every token has its own state” describes a logical correspondence. It doesn’t mean Prefill runs the input one token at a time, and it doesn’t mean every token gets its own GPU memory allocation.

As a goal for reuse, the sentence is fine. But the server has to decide whether it can reuse anything before it recomputes the KV. The common approach is to give the input prefix an identity that can be looked up quickly, such as a prefix tree or a hash, and then find the matching cached state.[2]

For a causal model with fixed weights and the same input representation, positions and attention rules, an identical prefix is a sufficient condition for exact reuse, and one that is easy to check. Real systems also have to consider the model or adapter identity, multimodal content, tenant isolation and similar conditions.[3]

A sufficient condition is not the only mathematically possible one. Different inputs can happen to produce the same intermediate values, and special models may have stronger state equivalences. But an ordinary prefix cache doesn’t need to prove every equivalence; it picks a lookup rule whose correctness is easy to guarantee.

A request can reuse just the first part of its input. Suppose a request has 110,000 input tokens in total: 100,000 of them come from an existing cache, and the remaining 10,000 have to be processed again. Counted by tokens, the hit rate is:

“The share of requests with any hit at all” and “the share of all input tokens that hit” are two different metrics. If each of a hundred requests hits just one token, the first can still be 100%, without necessarily saving much computation.

This also explains a common piece of engineering advice: as long as it doesn’t change what the task means, put stable material as early as possible, and put changing content, such as timestamps and this turn’s question, later. One dynamic field near the start can stop a long stretch of identical text after it from being reused directly.[4]

Compare “I like apples” with “I hate apples”. The last token is the same and sits in the same position, but when the model processes “apples”, it reads a different history. The representations produced by earlier attention layers become the input from which later layers compute K/V.

So using only token_id or (token_id, position) as a general cache key for higher-layer KV lumps different contexts together. The key also has to tell apart what came before.

But this does not yet show that “the cache key must contain the entire prefix in plain text”. That is exactly the next question.

Yes. Your parent-hash scheme works. A prefix’s identity can be represented recursively. Let h₀ encode the initial identity conditions, and for each token compute:

vLLM’s hash_block_tokens does exactly this: it hands the parent block’s hash, the current block’s tokens and extra identity information to the hash function together. That is the same idea as “parent_hash + current_token”, except that it usually works in blocks.[3]

The hash map you proposed can point to the real KV. The idea never required “replacing the entire model state with a few dozen bytes”. The key boundary is this: shortening the label is easy; letting the model keep working from only a small state needs a different computational structure.

Writing an equals sign between KV_B and Hash(A,B) confuses the two. The clear way to write it is cache[key] → KV, or a reference to a restorable state.

You can. SGLang’s public RadixCache code has a matching branch for page_size = 1; in the ordinary one-token-key mode, prefix matching is precise down to a single token. That is enough to show there is no general theoretical barrier to per-token indexing.[5]

A coarser granularity is usually a cost trade-off. Put a run of consecutive K/V into one block, and indexing, reference counting, transfer and eviction can all be handled as a group. In a simplified scheme that reuses only complete blocks, with a shared prefix of length p and a block size of b:

vLLM’s prefix caching documentation describes a scheme that reuses complete blocks. With b = 16, 999 identical tokens can be reused only up to 992 under that rule: 7 extra tokens are computed in exchange for fewer directory entries.[2]

But don’t go on to picture “per-token indexing” as “one GPU memory allocation per token”. The physical memory pool, the matching granularity and the way tree nodes are represented can each be designed separately, and a compressed prefix tree need not give every token its own tree node. Smaller physical pages can even reduce the unused space at the tail; the costs have to be measured on the actual system.

Da Wang: A short key saves the space of storing the entire prefix over and over, but the chain still has to be built in order. vLLM’s hash_block_tokens takes a parent_block_hash, so a new block’s key needs its parent’s key first. For a sequence of length N that is being indexed for the first time, with block size b, there are about N/b key constructions, one after another. Changing b from 16 to 1 makes that chain about 16 times as long.[3]

What is being compared here is the extra tokens at one divergence boundary against the indexing work for a whole new path. That is not enough to declare that “a constant saving can never beat a linear cost”: the attention cost of one token can also grow with the length of the history, and a cache may be reused by many requests.

Two more qualifications. Hashing still has to read the current block’s tokens, so the total work of scanning the input is usually still O(N); an existing session can continue incrementally from a parent hash it already has, so new indexing mainly corresponds to ΔN. Compressed trees, batching and different hashing schemes may also be implemented differently. So the serial chain should be seen as a concrete cost of this kind of implementation, not an inevitable flaw of every per-token cache.

Better ways to think of it are as a cache boundary or a restore point. Reading “breakpoint” as a cut-off point makes it easy to assume that whatever comes after it is no longer sent to the model.

Putting cache_control on a content block says this: the prefix from the start of the whole effective prompt up to the end of this block becomes a candidate for caching. Claude’s documentation sets the order as tools → system → messages.[4]

Producing the underlying K/V doesn’t wait for this marker. The API marker decides how the service layer stores, looks up and bills; it tells you nothing about when the GPU starts having K/V.

Several cache breakpoints can match different degrees of stability: tools and system instructions almost never change, project material changes now and then, and the conversation keeps growing. When a longer entry becomes invalid, a shorter one may still work. But each of them means “from the start up to here”; none turns identical text later on into a cache segment that stands on its own, apart from what came before.

It depends on how your requests change. Claude’s current documentation supports a top-level cache_control, which puts the cache breakpoint on the last cacheable content block. For requests that keep appending to an old conversation, that is convenient.[4]

In the second case, it is usually better to put an explicit cache breakpoint at the end of the stable material A. Automatic caching won’t find every stable segment for you and create entries for them. The current rule also looks back only so far: it checks at most 20 content-block positions around each boundary, and it looks for entries that were actually written before.[4]

response = client.messages.create(
    model=model_id,
    max_tokens=1024,
    cache_control={"type": "ephemeral"},
    system=stable_system,
    messages=growing_history,
)

response = client.messages.create(
    model=model_id,
    max_tokens=1024,
    system=stable_system,
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": stable_document,
             "cache_control": {"type": "ephemeral"}},
            {"type": "text", "text": current_question},
        ],
    }],
)

The code shows the shape of the request; your application has to supply client, model_id and the content variables. The prefix also has to reach the model’s minimum cacheable length. Caches expire; the current documentation offers 5-minute and 1-hour options. For how to call it, its limits and what is supported, follow the documentation of the platform you use.

When does the clock start?

Claude’s current documentation starts the TTL when the request that writes or reads the cache begins. For example, if a response streams for 4 minutes, a default 5-minute TTL has only about 1 minute left; if you then also wait for a tool to run, you may be too late to reuse the same entry.[4]

No error, yet usage is all zeros. Why?

A cached prefix has to reach the model’s minimum length. Some of the thresholds in the current official documentation are below. A request below the threshold is still processed normally; it does not raise an error just because nothing was cached.[4]

Recorded on September 10, 2026. The threshold applies to the length of the marked cacheable prefix, not just to whether the whole request is long enough. If cache_creation_input_tokens and cache_read_input_tokens are both 0, check this condition first; platform differences and model versions should still be checked against the corresponding documentation.

DeepSeek’s context caching documentation names three kinds of places where it persists state: the end of a request’s input and output, the common prefix of several requests, and fixed token intervals within long inputs or outputs. A later request can reuse a saved cache prefix unit only if it matches that unit completely.[6]

An abstract formula makes this clear. Let p be the length of the prefix two requests actually share, and E the set of boundaries that have been saved and can be restored. Under this complete-unit rule, the furthest position you can restore to directly is:

Request boundaries catch the natural points to continue from; common-prefix detection adds the reuse points that actually occur; fixed intervals give long sequences extra coverage. All of them add to the “states that can be found and used again”; none makes two prefixes that were different the same.

The diagram above explains the service’s behavior. DeepSeek does not publish the data structure of its full cache directory in this API documentation, so you can’t conclude from it that the server must use a particular radix tree, hash chain or checkpoint spacing.

Da Wang: Every extra entry you keep takes index space, write bandwidth and storage space, and some models also need extra restore snapshots. So something has to decide which states deserve to hold those resources. That is a cache admission policy.

A common prefix being used again is a stronger reuse signal than “it has appeared once”. When resources are limited, keeping such prefixes first reduces how much single-use content crowds out the hot cache. DeepSeek’s documentation also notes that building the cache takes a few seconds, so there may still be a delay between finishing the computation and the persisted entry becoming usable.[6]

“More worth saving once it is used again” is related to the idea of cache-on-second-access. But DeepSeek also saves entries at request boundaries and fixed intervals, so the whole system can’t be summed up as a strict second-access cache, and you can’t claim that every prefix must show up twice before it is saved to disk.

This goes deeper than “too many index entries”. In a standard implementation that keeps the full history of K/V in every layer, the idea works: as long as the data is still there, just take the first A positions. Whether the corresponding entry can be found is a separate question.

But many long-context architectures deliberately throw away old local state. DeepSeek’s documentation also says explicitly that sliding window attention changes how its cache is stored and matched.[6]

So “a compressed restore state exists for A+B” does not automatically imply “the full restore state of A can still be sliced out of it”. The memory structure a model uses directly affects how fine-grained a cache system’s restore points can be.

The same prefix can miss because the cache hasn’t finished writing, has been evicted or has expired, or because isolation conditions differ. DeepSeek’s current documentation describes a disk cache that is on by default, and says that building it takes time, that it works on a best-effort basis, and that it does not promise a 100% hit rate.[6]

The above is a conditional causal chain, not a measured result showing that a product’s hit rate went up. A smaller cache lets you keep more history, or lets you spend the resources on serving more concurrent requests; how they are finally allocated depends on the system’s policy.

At this point the hash has done what it is good at: identifying things quickly. The next question starts to touch the model architecture: why is the state that has to be kept so large?

In an ordinary full-history Transformer, each layer produces K/V from its own intermediate representation. Every layer has to remember the entire history, so the longer the sequence and the more layers there are, the bigger the cache.

This formula describes only a simplified shape of standard full-history K/V. It also makes it easy to see where several optimizations act: fewer KV heads, fewer dimensions per state, lower numerical precision, fewer kept positions, or fewer layers that each store their own long-term KV. YOCO mainly goes after the last one.[7]

The sixth item has to be achieved through architecture and training. In an ordinary model, K and V can differ, and the server cannot merge the two vectors into one and keep the original computation. Question 18 uses a sharing scheme with positional rotation to make this boundary clear.

YOCO stands for You Only Cache Once. The original paper splits the model into a first half, the Self-decoder, and a second half, the Cross-decoder. The first half produces the context representation, from which the shared global K/V is generated; each layer in the second half reads that memory with its own queries.[7]

“Once” means that only one set of global history K/V is kept, and several second-half layers reuse it. It doesn’t mean a token is computed only once in the whole model, it doesn’t mean the model runs only once across requests, and it doesn’t mean the first half has no state at all. The original paper makes that last point in a footnote.[7]

So YOCO does not squeeze a history of a million tokens into a fixed-length hash either. What it reduces is the repeated storage of long-term memory from layer to layer. The longer the context, the smaller the share taken by the fixed-size local windows.

The two halves still run in causal order: the current position can use only the history it is allowed to see, and each later new token passes through both halves again. Saying “the first half is like an encoder” does not mean it can read in both directions and see future tokens.[7]

In an ordinary Transformer, history positions must pass through the earlier layers to produce the K/V that each later layer uses. Even if you don’t need those history positions’ outputs, it’s hard to skip that computation.

YOCO changes the dependencies: the history K/V that the second-half layers need comes directly from the first half’s output. So history positions that are already given no longer have to pass through the whole second half just to build a cache for later use. In the public implementation, CrossDecoder.forward first generates and appends the shared K/V, then uses skip_cross_decoder to decide whether to keep running the second-half layers.[8]

This design has to be understood position by position. If the prompt has N positions, getting the first new token’s probabilities still means running the second half on the last input position’s representation. What can be saved is the second-half computation for many earlier history positions.

Beyond that, using efficient local or recurrent structures in the first half also lowers the cost of processing long inputs. The speedups in the paper depend on the baseline model, the sequence length and the implementation; you can’t paste them straight onto another product.

It can be spelled out as: which layers produce history, which layers share it, and which state each layer still keeps for itself. Following the configuration in the technical excerpt, the 40 layers split into two parts, 0–19 and 20–39; the global KV sources are [2, 8, 14, 20], and the first two layers keep only a sliding window.[9]

The excerpt ties the second half’s global KV to the first half’s final representation, while each layer keeps its own local sliding-window state. That has a clear structural link to YOCO’s cross-layer memory sharing; inside the first half, layers can also share in groups. When comparing models, look at the actual dependency edges, not just the two names “Encoder / Decoder”.[7][9]

First, be clear about the unit: for each extra original input token, how much more global KV has to be kept. That is different from “how much GPU memory the whole sequence takes”. Under the given quantization and sharing conditions, each compressed position needs 356 bytes.[9]

The scales here are the factors needed to bring low-precision numbers back to their proper size. In FP4 each number takes 0.5 bytes, but you can’t just multiply by 0.5 and stop: the main KV’s scales add 32 bytes, and the indexer K’s scales add another 4. The report excerpt calls them E4M3 and UE8M0 scales respectively.[9]

Now the local windows: estimating with 40 layers, 128 positions per layer, 512 dimensions and an FP8 carrier at 1 byte per number, the numerical payload of SWA is:

K and V describe two uses: one takes part in matching, the other in summing up. A model can be designed so that the same stored vector serves both uses; then one fewer set of numbers has to be stored. The technical excerpt says V4.1’s kv_shared is used for both matching and the output multiplication, and the ledger is calculated on that condition.[9]

Rotary position encoding (RoPE) rotates part of a vector according to the token’s position, so that matching can express relative position. If the rotated vector used as the Key is also the Value, the summed output carries those rotations too. Multiplying the output for query position i by R_i⁻¹ expresses it in the current position’s coordinates.

Likewise, num_key_value_heads = 1 by itself only says there are few KV heads; it cannot on its own prove that K and V are the same vector. Whether they are shared requires looking further at the actual projections, the tensor references, and the inputs of the two matrix multiplications.

Look at it as two questions: how to bring back the local state that wasn’t saved, and how exact the restored state needs to be. A single layer looks only at the most recent W positions, but once layers are stacked, the theoretical range of dependence usually widens. For a simple stack of L layers whose windows include the current position, the receptive field of a single output position can reach 1 + L(W−1).

The technical excerpt describes V4.1’s SWA Bounded Replay like this: to avoid saving every layer’s SWA state, it replays only one window to restore the state approximately. The excerpt also notes that the report admits the actual effective dependence may be shorter than the theoretical receptive field, and lists this kind of restoration as a robustness boundary that needs further characterization.[9]

Da Wang: It does. YOCO’s visible reference implementation has skip_cross_decoder, which can follow the shared-KV dependencies to skip the second-half computation for many history positions. The V4.1 technical excerpt, on the other hand, separates two paths:

So getting from “the architecture allows a short path” to “the server is actually faster” still goes through the scheduling implementation and measurement. The last input position has to produce the probabilities of the first new token; scoring every position, or training, can’t directly use the skip that works when you only generate the next token.

One more question decides speed: for every generated token, how much history state does it actually have to touch? Squeezing all the KV from 10 GB to 1 GB reduces the pressure on capacity; if many layers still scan the full history again and again, reading and matching stay expensive.

According to the technical excerpt, layer 20 first builds a candidate pool; several later indexers pick the top 512 only within that pool, and some other layers reuse the selection. That removes the repeated work of “every layer searching the whole history once”.[9]

Sparse reading also trades off capability: if a relevant position doesn’t make it into the candidate pool, the later fine ranking usually can’t conjure it back. So you have to look at candidate recall, answer quality and real latency together. Shared KV solves “how many copies to store”, shared candidates solve “how many times to search”, and sparse reading solves “how much to read each time”.

Engram gives the model an ability to “read a learned vector when a certain local combination shows up”. The official public implementation first maps an n-gram to several hash addresses, fetches the embedding vectors, and then injects them into the backbone network through a projection, a gate controlled by the current hidden state, and local fusion.[10]

An n-gram here means a local combination of n consecutive tokens. Once the model and table size are fixed, each position reads only a limited number of table entries; the addresses depend only on local tokens, so they can be prepared before the corresponding deep-layer computation happens, which makes them suitable for prefetching. The official paper also discusses a system arrangement that keeps large tables in host memory.[11]

“Store every n-gram” also needs a qualification: the hash table has limited capacity, and it does not give every possible combination its own collision-free slot. Different combinations can land at the same address, and multi-head hashing and training together deal with that compression. Unlike an exact KV cache, which tries hard to avoid mismatched identities, here collisions are a condition the representation design itself has to face.[10]

The V4.1 technical excerpt lists 2/3/4-gram memory used at layers 1 and 14. That describes one particular assembly; the original Engram demo code has a different configuration. Whichever layers it is placed in, it does not get around the fact that “different context can mean different higher-layer KV”: once static table entries are fused with dynamic hidden states, they can still produce different results.[9][10]

So the “memory” inside a model includes at least two kinds of things: what this request computed in the past, and what the training stage learned for common patterns. One has its lifetime managed by the cache system; the other usually travels with the model weights. The two can exist side by side.

First, map out the real input order of every request and mark the earliest position that changes. For append-only conversations, keep existing content in its original order as far as possible; for different questions about fixed material, separate the stable material from the dynamic suffix, and place cache boundaries according to the service’s rules.

Then look at the actual usage, and tell apart “cache reads”, “cache writes” and “regular input”. Don’t assume that fields called input_tokens at different providers are the same kind of total.

read  = usage.cache_read_input_tokens
write = usage.cache_creation_input_tokens
fresh = usage.input_tokens

total_input = read + write + fresh
hit_ratio = read / total_input if total_input else 0

In Claude’s documentation, this input_tokens does not include the cache-read and cache-write parts above. Server-side tools and multi-turn aggregation should also be checked against the corresponding usage structure.[4]

hit  = usage["prompt_cache_hit_tokens"]
miss = usage["prompt_cache_miss_tokens"]

hit_ratio = hit / (hit + miss) if hit + miss else 0

The field definitions come from DeepSeek’s context caching documentation. For batch statistics, use “sum of all hit tokens / sum of all input tokens”; hit percentages of requests with different lengths shouldn’t simply be averaged.[6]

Then put cost and acceptance back in the same table. As a pure input-price model that ignores the write premium, with input volume N and hit rate h:

A more valuable final metric is total cost during the evaluation ÷ the number of tasks that pass acceptance, reported together with the success rate, the task types, and the median and slow tail of the time taken. Measure a cold cache and a warmed-up cache separately, so that different conditions aren’t blended into one good-looking average.

For schemes with approximate restoration or sparse reading, evaluation should also cover consistency before and after restoration, task failures caused by missed candidates, and how performance accumulates over long sessions. The optimization target is always to pass the same acceptance checks; the bytes and scans saved have to show up in the end as task cost and time to completion.