---

Abstract #

Decoder-only language models keep a key-value (KV) cache: the key and value vectors of every token read so far, reused at each generation step. A common worry is that the cache goes stale when later text changes what an earlier word means, as in “The bank” followed by “of the river” or “approved the loan”. We ask whether the cache is exact, where re-interpretation happens, and where exactness ends. We ran five CPU experiments on three Pythia models (70M to 410M parameters, causal) and BERT-base (bidirectional), using 16 ambiguous words, 2 garden-path sentences and 24 Pile test documents. First, the cache is exact up to rounding: at 64-bit precision, decoding with and without it gave identical text for 24 of 24 prompts (maximum logit difference 2.54659e-11), and an ambiguous word’s cached keys and values were bit-identical under both continuations in 18 of 18 items, while BERT’s state for the same word moved by a cosine distance of 0.529898. Second, meaning is resolved at new positions: a sense probe at the old cached position stays at chance (50%) in every layer, while the word repeated after the context reaches 90.625%. Third, exactness ends where engineering intervenes: top-1 agreement with the exact cache is 86.30% for a bf16 cache, 30.76% for int4, 81.52% for a 128-token window with four attention-sink tokens and 60.57% without, and 91.11% for a document chunk cached outside its context.

1. Introduction #

Our previous article in this series, KV-Cache Compression Techniques in Production: Cost Savings vs Quality Degradation Trade-offs[1]^[2], priced what quantising and evicting the KV cache saves and what it costs in quality. It took the cache itself for granted. This article returns to the mechanism and a question engineers often ask about it: if a model stores its reading of “bank” and then learns three words later that the bank is a river bank, is the stored reading now wrong? And if it is not wrong, how can a model whose past states never change still change its mind about what a sentence means?

The answer matters for cost and correctness. Prefix caching, paged memory and provider prompt caching all rest on the claim that a cached entry is as good as a recomputed one. Where that holds, caching is a free saving; where it fails, it silently changes outputs.

RQ1: Is the KV cache of a causal decoder an exact substitute for recomputation when later tokens change the meaning of earlier ones, and how does a bidirectional encoder differ? RQ2: If cached entries never change, where in the network does re-interpretation of an earlier word happen? RQ3: Which common cache operations (lower precision, quantisation, window eviction, reuse of a chunk computed in another context) break exactness, and by how much?

For the Cost-Effective Enterprise AI series, the answers separate cache optimisations that are free from those that trade quality for money, which is the distinction every serving budget depends on.

2. Existing Approaches (2025–2026 state of the art) #

2.1 Causal attention: why the past cannot see the future #

In a transformer, each layer turns every token’s hidden state into a query, a key and a value. A token’s new state is a weighted sum of the values of the tokens it attends to, with weights from the dot products of its query with their keys. Decoder-only language models apply a causal mask: the original transformer masks out all positions after the current one inside scaled dot-product attention, to “prevent leftward information flow” and keep generation auto-regressive[2]^[3]. The consequence is structural. The hidden state of token i in layer 1 depends only on tokens 1 to i; by induction, so does its state in every later layer, and therefore so do the key and value it contributes to each layer. Appending tokens cannot change them. Caching them is not a heuristic: it stores a function of the prefix that later tokens are not allowed to influence.

Generation then splits into two phases. Prefill runs the whole prompt in parallel and writes its keys and values; decode runs one new token at a time, computes that token’s query, key and value, appends the new key and value, and attends over everything stored.

flowchart LR
    P[Prompt tokens] --> F[Prefill: all positions in parallel]
    F --> C[KV cache: keys and values per layer]
    N[New token] --> Q[Its query, key, value in each layer]
    Q --> A[Attend over all cached keys]
    C --> A
    A --> H[New hidden state and next-token logits]
    Q -->|append, never overwrite| C
    H --> N

2.2 Bidirectional encoders and diffusion models: where caching breaks #

BERT was designed the other way round: it pre-trains “deep bidirectional representations” by conditioning on left and right context in all layers[3]^[4]. Every token’s state, and therefore every key and value from the second layer on, depends on what comes after it. A BERT-style cache would be wrong as soon as a token is appended. The problem has returned with diffusion language models, which denoise many positions at once under bidirectional attention. BiCache reports that because updating any token alters the keys and values of the entire context, applying LLM prefix-caching techniques to diffusion models makes accuracy “collapse to near zero”; it restores reuse only for a dynamically chosen depth of shallow layers, where shared-prefix states stay stable[4]^[5]. Fast-dLLM introduces a block-wise approximate KV cache for bidirectional models with up to 27.6 times higher throughput[5]^[6], and Elastic-Cache observes that cached states drift more in deeper layers and refreshes those selectively[6]^[7]. For these models the cache is a managed approximation; for causal decoders it is exact by construction.

2.3 Positions and prefix caching #

Most current decoders encode position with rotary position embeddings (RoPE), which rotate queries and keys by an angle proportional to position so that their dot product depends on the relative distance[7]^[8]. The rotation is applied before the key is cached, so a cached key carries its absolute position. That is harmless when the prefix is reused in place and becomes a problem when a cached chunk is moved (section 2.4).

Serving systems exploit exactness across requests. PagedAttention stores the cache in fixed-size blocks, like virtual-memory pages, with near-zero wasted memory and sharing of blocks within and across requests, which raised throughput 2 to 4 times[8]^[9]. vLLM’s automatic prefix caching identifies each full block by a hash of its own tokens together with all the tokens before it, and its documentation describes the technique as one that “won’t change model outputs”[9]^[10]. The hash over the whole prefix is the causal mask turned into a cache key: a block is reusable exactly when everything it could have attended to is the same. LMCache moves such caches out of GPU memory and shares them across engines, reporting up to 15 times higher throughput with vLLM on multi-round question answering and document analysis[10]^[11].

Exact reuse has two practical edges. Shared caches leak information through timing: an audit detected global cache sharing across users at seven API providers[11]^[12], and PrefixWall isolates only suspicious prefixes, keeping up to 70% more cache reuse than per-user isolation[12]^[13]. And because any change early in the prompt invalidates everything after it, prompt layout decides the hit rate: in long agentic sessions, caching cut API costs by 41–80%, with the most consistent gains when dynamic content is placed at the end[13]^[14]. The hub’s analysis of prompt-caching return on investment covers the economics of that discount[14]^[15].

2.4 Reusing chunks outside their original context #

Retrieval-augmented generation (RAG) wants to reuse a document’s cache wherever the document appears, not only as a prefix. Such a cache is exact for the context in which it was computed and wrong elsewhere, because it never attended to the new preceding text. CacheBlend recomputes the keys and values of a small subset of tokens to restore that cross-attention and reports a 2.2 to 3.3 times faster time to first token[15]^[16]. A 2026 testbed finds that naive segment reuse largely preserves quality for single-document question answering and summarisation but shows clear limits in cross-document reasoning, while lifting the token-level hit rate from 3.3% to 63.9% under a RAG-style workload[16]^[17]. Another line retrains the model with an encoder so that chunk caches are position-independent by design, cutting time to first token by 51–94%[17]^[18]. The same context-dependence is a security risk: HijackKV points out that a cache retrieved by token match still encodes the context in which it was originally computed, and uses this to hijack reused chunks with a 94% average success rate in a single attempt[18]^[19].

2.5 Eviction, attention sinks and windowed layers #

When the cache is too large, entries are dropped. StreamingLLM showed that a plain sliding window fails once text exceeds the window, and that keeping the first few tokens recovers most of the loss because models park large attention weights on them as an “attention sink”[19]^[20]. A 2025 analysis argues that sinks let a model avoid over-mixing information across tokens[20]^[21], and a sigmoid gate after attention mitigates them[21]^[22]. Model designers now build eviction in: gpt-oss alternates dense layers with 128-token banded-window layers and gives each head a learned softmax bias that lets it attend to nothing[22]^[23], and Gemma 3 uses five local layers with a 1,024-token span per global layer to contain KV memory[23]^[24]. Quantisation formats and their measured costs are covered in our previous article[1]^[2]; a 2026 survey classifies more than thirty cache-management systems by locality, lifetime, ownership and storage substrate[24]^[25].

flowchart TD
    K[Operations on a KV cache] --> E[Exact up to rounding]
    K --> X[Approximate]
    E --> E1[Decode with causal mask]
    E --> E2[Prefix caching keyed on full prefix]
    E --> E3[Paged blocks shared across requests]
    E --> E4[Offload and reload unchanged bytes]
    X --> X1[Lower precision or quantisation]
    X --> X2[Eviction and sliding windows]
    X --> X3[Chunk reuse outside its context]
    X --> X4[Bidirectional or diffusion models]
    X3 --> R[Partial recomputation, e.g. CacheBlend]

3. Method and Data #

Models. We used three checkpoints of EleutherAI’s Pythia suite (70M, 160M and 410M parameters), decoder-only models with rotary embeddings trained on the Pile[25]^[26], and bert-base-uncased as the bidirectional contrast[3]^[4]. Everything ran on a six-core CPU in 32-bit floating point (fp32) unless stated; to separate rounding from real differences, the key exactness tests were repeated in 64-bit (fp64).

Data. We wrote 16 ambiguous English nouns (bank, bat, pitcher, mouse, spring, seal, plant, match, organ, pen, key, bark, cell, club, fan, palm), each with two continuations that force different senses after the prefix “The word“, plus three unambiguous reference sentences per sense. Two garden-path prefixes (“The old man”, “The horse raced past the barn”) were added for the exactness test. Every word is a single token in both tokenisers. For generation and eviction tests we used 24 documents from the Pile test split, of which 16 are at least 512 tokens long.

Experiments. E1 compares greedy decoding of 64 tokens from 32-token prompts with the cache and with full recomputation at every step. E2 computes the keys, values and hidden states of a prefix alone and inside each of its two continuations. E3 builds per-layer sense centroids from the reference sentences and classifies the word’s state by the nearer centroid at three places: the original cached position, a new position where the word is repeated after the continuation, and the original position in BERT; it also records how much attention the new position pays to the old word. E4 times one decode step with and without the cache and computes cache bytes per token for public model configurations, checked against each model’s published configuration. E5 decodes 256 teacher-forced tokens after a 256-token prefill with a modified cache, measuring the Kullback-Leibler (KL) divergence and top-1 agreement against the exact cache, and perplexity. The modifications are casting to fp16 or bf16, naive symmetric int8/int4/int2 quantisation with one scale per token and head, a 128- or 64-token window with or without four sink tokens, and a 128-token chunk whose cache was computed without its preceding 128 tokens, either at its original positions 0–127 or re-indexed to 128–255.

All code, data and outputs are in stabilarity/hub/research/how-the-kv-cache-works-why-cached-tokens-stay-valid-when-later-tokens-change-their-meaning.

4. Results #

4.1 RQ1: The cache is exact; the encoder’s would not be #

Decoding with the cache reproduced decoding without it. In fp32 the generated sequences were identical for 24 of 24 prompts on Pythia-70M and Pythia-410M and for 23 of 24 on Pythia-160M; the largest logit differences were 0.0470581, 0.0119629 and 0.00633657, against median logit standard deviations of 64.3335, 80.5287 and 3.25238. Repeating the Pythia-160M test in fp64, all 24 sequences matched and the largest logit difference fell to 2.54659e-11. Across every run the argmax of the cached logits agreed with full recomputation on 100% of positions. The residual differences are those of floating-point arithmetic that is not associative, the same mechanism that makes greedy output depend on batch size and GPU count in production[26]^[27].

The state of an ambiguous word does not move when its meaning is decided later. Across the 18 prefixes, the keys and values Pythia-160M stores for the prefix were bit-identical under the two continuations in 18 of 18 cases in fp32 (17 of 18 in fp64, with the remaining difference at 3.51719e-13). Comparing the prefix computed alone with the prefix inside a longer sequence, the largest difference in a cached key was 0.000640869 in fp32, against cached values as large as 63.3572, and 1.6982e-12 in fp64; matrix kernels sum in a different order for different sequence lengths, which accounts for the fp32 residue.

BERT behaves as the theory predicts. The same word’s state in the two contexts is identical at the embedding layer and diverges with depth, reaching a cosine distance of 0.14955 after the first layer and 0.529898 after the twelfth (Chart 1). The keys BERT would have to cache change too: relative to their norm, they differ by 0.58184 between the two contexts in the second layer and by 0.998699 in the last. A cache built while reading “The bank” would be wrong for every layer above the first once “of the river” arrives, which is the BiCache problem[4]^[5] in miniature. In Pythia, the cosine distance stays below 2e-15 in every layer.

4.2 RQ2: Meaning is resolved at new positions, by reading the old ones #

If the cached state of “bank” never changes, it cannot carry the river sense. Chart 2 confirms this and shows where the sense goes instead.

At the original cached position, Pythia’s sense accuracy is exactly 50% at all 13 depths (embeddings plus 12 layers). That is chance by construction: the state is the same for both senses, so it is classified correctly for one and wrongly for the other. When the word appears again after the disambiguating text, its new state carries the sense: accuracy rises from 50% at the embedding to 71.875% after the first layer and 87.5% after the second, peaking at 90.625% in layers 5, 6 and 12. BERT reaches 100% at the original position from the first layer on, because there the old position itself is rewritten.

The attention weights show how the new position gets its information. Averaged over heads, the repeated word puts between 0.506078 and 0.677254 of its attention weight on the disambiguating continuation in every layer, and up to 0.247605 on the first token, the attention sink[19]^[20]. It pays the old copy of the word itself little attention: a weight of 0.117683 in the second layer, below 0.014 from the fifth layer on, and almost none in the last two. The “re-interpretation” of an earlier word is therefore a computation performed by later positions: their queries read the unchanged keys and values of the whole context, and their own deeper-layer states hold the combined meaning. This is why decoder models used as text encoders benefit from seeing the input twice: sequence repetition makes token embeddings effectively bidirectional and surpasses both encoders and decoders with the mask removed[27]^[28], and Causal2Vec reaches a similar effect by prepending one contextual token from a small bidirectional encoder[28]^[29]. In generation, every new token is such a second look.

flowchart LR
    T1[The] --> T2[bank: state fixed when read]
    T2 --> T3[of the river ...]
    T3 --> T4[next token]
    T4 -->|query reads keys and values| T2
    T4 -->|query reads keys and values| T3
    T4 --> M[Deeper layers of the new position hold the river sense]

The fixed past is also what makes deliberate edits possible: cache steering adds a one-shot vector to cached keys and values to change a frozen model’s behaviour[29]^[30]. Ordinary inference never does this.

4.3 RQ3: Where exactness ends, and what it buys #

Exactness is the default, and it pays. On our CPU, one decode step of Pythia-160M took 39.741 ms with the cache and 110.86 ms without it at 128 tokens of context, and 48.505 ms versus 1430.22 ms at 2,000 tokens, a 29.49-fold speed-up (Chart 3). The analytical compute ratio is larger, 127.811 at 128 tokens and 1771.51 at 2,000, because a small model on a CPU spends much of each step on fixed overheads.

The price is memory. Table 1 and Chart 4 apply the standard formula, two tensors times layers times KV heads times head dimension times two bytes, to published configurations.

Model	Attention	Layers	KV heads x head dim	KV cache per token (KiB, 16-bit)	Per 128K-token sequence (GiB)
Pythia-160M	MHA	12	12 x 64	36	4.5
Llama-3.1-8B, one KV head per query head	MHA	32	32 x 128	512	64
Llama-3.1-8B	GQA	32	8 x 128	128	16
Qwen3-32B	GQA	64	8 x 128	256	32
Llama-3.1-70B	GQA	80	8 x 128	320	40
DeepSeek-V3	MLA, 512 + 64 per layer	61	latent	68.625	8.57812

Source: configurations from the Pythia, Llama 3, Qwen3 and DeepSeek-V3 reports[25]^[26][30]^[31][31]^[32][32]^[33], checked against the models’ public configuration files; our calculation. MHA: multi-head, GQA: grouped-query, MLA: multi-head latent attention. The first Llama row is a counterfactual.

Grouped-query attention cuts the cache four-fold at the same depth, and DeepSeek-V3’s latent attention caches one 512-dimensional latent and a 64-dimensional positional key per layer[32]^[33]. These are architectural choices made before training, and the cache remains exact for the model as trained. Shrinking the cache after training is different, and Chart 5 and Table 2 show what each such operation does to the next-token distribution.

Cache operation, Pythia-160M	Mean KL vs exact (nats)	Top-1 agreement vs exact (%)	Perplexity
Exact fp32 cache	0	100	16.9647
fp16 cache	0.00224419	97.36	16.9871
bf16 cache	0.064307	86.30	17.8661
int8, per token and head	0.144177	81.18	20.007
int4, per token and head	2.29758	30.76	185.136
int2, per token and head	4.8392	10.06	2516.06
Window 128 incl. 4 sink tokens	0.233131	81.52	21.3771
Window 128, no sinks	0.701864	60.57	35.0835
Window 64 incl. 4 sink tokens	0.403609	74.66	25.2434
Window 64, no sinks	0.991164	55.54	47.7408
Chunk cached alone, re-indexed positions	0.0423106	91.11	n/a
Chunk cached alone, original positions	0.204141	82.13	n/a

Source: our analysis, 16 Pile test passages, 256-token prefill plus 256 teacher-forced tokens; chunk rows use 128 + 128 + 64 tokens. Perplexity is on the actual next tokens.

Three patterns stand out. First, precision matters more than its byte count suggests for this small model: fp16, with 10 mantissa bits, keeps 97.36% agreement, while bf16, with 7, keeps 86.30%, and naive int8 81.18%. Pythia-160M’s logits are unusually large (median standard deviation 80.5287), so small perturbations move the argmax; outlier-aware schemes such as those compared in our previous article do far better than one scale per token[1]^[2]. Second, eviction costs depend on sinks: four retained initial tokens lift a 128-token window from 60.57% to 81.52% agreement and cut perplexity from 35.0835 to 21.3771, reproducing the StreamingLLM effect[19]^[20] on Pythia. Third, a chunk computed without its context is the mildest error in the table once its positions are corrected (91.11% agreement, KL 0.0423106), but leaving the rotary positions of its original computation in place more than doubles the agreement loss to 82.13%. Missing cross-attention is the irreducible error that CacheBlend-style recomputation targets[15]^[16]; stale positions are an avoidable one.

5. Discussion #

How meaning stays agile. The causal decoder never revises its past. It keeps an append-only record of what each token meant given what came before it, and it re-reads that record from every new position. Re-interpretation is a property of the reader, not of the record: the query of each new token decides how to weight the old keys, and the new token’s deeper layers combine them into a state that reflects the whole context. An old token’s representation stays a correct answer to a narrower question, “what did this token mean given only its left context”, and later tokens never need it to answer anything else. Bidirectional models answer the wider question at every position, which is why they must recompute and why diffusion models now need explicit cache-refresh policies[6]^[7].

Implications for practitioners. Treat exact operations and approximate ones differently. Prefix caching, paged sharing and offloading of unchanged bytes are free in quality and should be maximised by putting stable content first and variable content last[13]^[14]. Precision changes, quantisation, eviction and non-prefix reuse need an acceptance test on the target model, because their cost varies with the model and the workload.

flowchart TD
    S[Proposed cache change] --> Q1{Same tokens, same positions, same bytes?}
    Q1 -->|Yes| A[Exact: prefix cache, page, offload]
    Q1 -->|No| Q2{Content computed in another context?}
    Q2 -->|Yes| B[Fix positions, recompute selected tokens, check isolation]
    Q2 -->|No| Q3{Entries dropped?}
    Q3 -->|Yes| C[Keep sink tokens, measure on own workload]
    Q3 -->|No| D[Lower precision: measure KL and task score]

Limitations and threats to validity. Our models are small (70M to 410M parameters) and run on CPU; the size of each approximation error will differ on billion-parameter models, which tolerate precision loss better, and on GPU kernels. The quantiser is deliberately naive, so it bounds the error from above rather than representing production formats. The sense probe uses 16 hand-written words and nearest-centroid classification with three reference sentences per sense, so the accuracy levels are indicative; the exact 50% at the old position does not depend on the probe, since it follows from identical states. The window variants keep original positions rather than re-indexing the cache as StreamingLLM does, and the chunk-reuse test covers one chunk length.

6. Conclusion #

RQ1 Finding: The KV cache of a causal decoder is an exact substitute for recomputation, limited only by floating-point rounding, while a bidirectional encoder’s cache would be invalid. Measured by identical greedy outputs in fp64 = 24 of 24 prompts (maximum logit difference 2.54659e-11), bit-identical cached keys and values under the two disambiguating continuations = 18 of 18 items, and BERT’s final-layer cosine distance for the same word = 0.529898. This matters for the series because prefix caching, paging and offloading can be adopted as pure cost savings with no quality budget.

RQ2 Finding: Re-interpretation happens at new positions, whose queries read the unchanged cache and whose deeper layers hold the updated meaning. Measured by sense accuracy at the old cached position = 50% at all 13 depths versus a peak of 90.625% at a new position after the context, with an attention weight of 0.506078 to 0.677254 from the new position on the disambiguating text. This matters for the series because it explains why prompt order, not cache freshness, determines what a model can use, and why stable content belongs at the front of a prompt.

RQ3 Finding: Exactness ends with lower precision, quantisation, eviction and out-of-context chunk reuse, and the errors are large enough to need testing. Measured by top-1 agreement with the exact cache = 86.30% for a bf16 cache, 30.76% for naive int4, 81.52% versus 60.57% for a 128-token window with and without sink tokens, and 91.11% versus 82.13% for a reused chunk with corrected versus stale positions, against a measured decode speed-up of 29.49 times at 2,000 tokens for the exact cache. This matters for the series because the cheap, exact savings should be exhausted before any of the approximate ones are bought with quality.