Skip to content

Stabilarity Hub

Menu
  • Home
  • Research
    • Healthcare & Life Sciences
      • Medical ML Diagnosis
    • Enterprise & Economics
      • AI Economics
      • Cost-Effective AI
      • Spec-Driven AI
    • Geopolitics & Strategy
      • Anticipatory Intelligence
      • Future of AI
      • Geopolitical Risk Intelligence
    • AI & Future Signals
      • Capability–Adoption Gap
      • AI Observability
      • AI Intelligence Architecture
      • AI Memory
      • Trusted Open Source
    • Data Science & Methods
      • HPF-P Framework
      • Intellectual Data Analysis
      • Reference Evaluation
    • Publications
      • External Publications
    • Robotics & Engineering
      • Open Humanoid
      • Open Starship
      • Living NPCs
    • Benchmarks & Measurement
      • Universal Intelligence Benchmark
      • Shadow Economy Dynamics
      • Article Quality Science
  • Tools
    • Healthcare & Life Sciences
      • ScanLab
      • AI Data Readiness Assessment
    • Enterprise Strategy
      • AI Use Case Classifier
      • ROI Calculator
      • Risk Calculator
      • Reference Trust Analyzer
    • Portfolio & Analytics
      • HPF Portfolio Optimizer
      • Adoption Gap Monitor
      • Data Mining Method Selector
    • Geopolitics & Prediction
      • War Prediction Model
      • Ukraine Crisis Prediction
      • Gap Analyzer
      • Geopolitical Stability Dashboard
    • Technical & Observability
      • OTel AI Inspector
    • Robotics & Engineering
      • Humanoid Simulation
      • Living Village (AI NPC game)
      • Wizard Village (AI NPC 3D game)
    • Benchmarks
      • UIB Benchmark Tool
    • Article Evaluator
    • API Gateway
  • EKIT Department
  • About
    • Contributors
  • Contact
  • Join Community
  • Terms of Service
  • Login
  • Register
Menu

KV Cache Quantization Benchmark: 4-Bit KIVI Finds 30 of 30 Needles

Posted on November 17, 2026November 17, 2026 by
Cost-Effective Enterprise AIApplied Research · Article 45 of 45
By Oleh Ivchenko

KV Cache Quantization Benchmark: 4-Bit KIVI Finds 30 of 30 Needles

Academic Citation: Ivchenko, Oleh (2026). KV Cache Quantization Benchmark: 4-Bit KIVI Finds 30 of 30 Needles. Research article: KV Cache Quantization Benchmark: 4-Bit KIVI Finds 30 of 30 Needles. Odessa National Polytechnic University, Department of Economic Cybernetics.
DOI: 10.5281/zenodo.24295214[1]  ·  View on Zenodo (CERN)
DOI: 10.5281/zenodo.24295214[1]Zenodo ArchiveSource Code & DataCharts (5)ORCID
4,863 words · 63% fresh refs · 3 diagrams · 36 references

86stabilfr·wuedophcgmx
BadgeMetricValueStatusDescription
[s]Reviewed Sources6%○≥80% from editorially reviewed sources
[t]Trusted94%✓≥80% from verified, high-quality sources
[a]DOI81%✓≥80% have a Digital Object Identifier
[b]CrossRef8%○≥80% indexed in CrossRef
[i]Indexed75%○≥80% have metadata indexed
[l]Academic83%✓≥80% from journals/conferences/preprints
[f]Free Access100%✓≥80% are freely accessible
[r]References36 refs✓Minimum 10 references required
[w]Words [REQ]4,863✓Minimum 1,000 words of article body. Current: 4,863
[u]Original [REQ]100%✓≥85% unique sentences (no padding or repeated blocks) and no leaked drafting notes. Current: 100% unique
[e]Evidence [REQ]31✓Cites at least 1 external source(s): links, DOIs or a reference list. Current: 31
[d]DOI [REQ]✓✓Zenodo DOI registered for persistent citation. DOI: 10.5281/zenodo.24295214
[o]ORCID✓✓Author ORCID verified for academic identity
[p]Peer Reviewed—○Peer reviewed by an assigned reviewer
[h]Freshness63%✓≥60% of references from 2025–2026. Current: 63%
[c]Data Charts5✓Original data charts from reproducible analysis (min 2). Current: 5
[g]Code✓✓Source code available on GitHub
[m]Diagrams3✓Mermaid architecture/flow diagrams. Current: 3
[x]Cited by0○Referenced by 0 other hub article(s)
Score = Ref Trust (82 × 60%) + Required (4/4 × 30%) + Optional (5/7 × 10%)

Abstract #

A KV cache quantization benchmark should answer one practical question: if you store the attention cache in 4 or 2 bits, or throw half of it away, does the model still find a fact buried in a long prompt? We measured this on two real open models on an ordinary 6-vCPU server: Qwen2.5-0.5B-Instruct and Qwen3-0.6B. The test is a passkey needle-in-a-haystack: a five-digit number hidden at 5%, 25%, 50%, 75% or 95% depth in prompts of 1,024 and 2,048 tokens. We compared 15 cache treatments. The baseline is a BF16 cache. Six are simulated INT8, INT4 and INT2 quantization, either per token or in the KIVI layout (keys per channel, values per token). Eight evict tokens at a 50% or 25% budget: a sliding window, StreamingLLM attention sinks, an H2O-style heavy-hitter cache and a SnapKV-style question-aware cache. We also measured WikiText-2 perplexity after a compressed 1,024-token prefix. Three numbers stand out. First, KIVI-layout INT4 retrieved the passkey in 30 of 30 prompts across both models at 31.25% of BF16 bytes, while per-token INT4 managed 20 of 30 and per-token INT2 none at all. Second, KIVI INT2 kept 27 of 30 at 18.75% of the bytes. Third, eviction depends on what is kept, not how much: a 25% sliding window retrieved 6 of 30, and a 25% SnapKV cache retrieved 30 of 30 when the question was known before compression. Quantization here is simulated, so these numbers measure accuracy, not kernel speed.

1. Introduction #

Our previous benchmark on the same server found that Qwen3-0.6B stores 112 KiB of key-value (KV) cache per token, 9.3 times as much as Qwen2.5-0.5B, and closed by asking whether KV cache compression could shrink that bill [1][2]. This article answers that question with measurements on the models themselves.

There are two families of answers. Quantization keeps every token but stores each number in fewer bits. KIVI, for example, reports near-baseline quality at 2 bits with 2.6 times less peak memory [2][3]. Pruning, also called eviction, keeps full precision but drops tokens. Both are usually evaluated on long-context suites with big GPU models. Readers searching for a KV cache quantization benchmark or for KV cache pruning tend to want something simpler: a controlled test on a model they can run, which shows where compression breaks.

The needle-in-a-haystack (NIAH) test is a good probe for this. It hides one fact in filler text and asks for it back [3][4]. Our earlier survey of long-context retrieval benchmarks explains why NIAH became the standard first check and where it falls short [4][5]. RULER showed that models which ace the plain NIAH test can still fail harder long-context tasks [5][6]. If the plain NIAH test breaks under compression, then, the harder tasks will not survive either. That makes NIAH a useful floor for any compression method.

RQ1: KV cache quantization benchmark: how much passkey retrieval accuracy and perplexity do INT8, INT4 and INT2 caches lose, and does it matter whether keys are quantized per token or per channel? RQ2: KV cache pruning: at a 50% and a 25% token budget, how do a sliding window, attention sinks, heavy hitters and a question-aware selector compare, and at which needle depths do they fail? RQ3: Per byte of KV memory, which compression method keeps the most retrieval accuracy, and what does it save in decode latency on a CPU?

What is KV cache (and why it eats memory) #

A transformer generates text one token at a time. Each new token attends to every earlier token through two vectors per layer and head: a key (used to decide where to look) and a value (the content that is read). Recomputing these for the whole prompt at every step would be wasteful, so inference libraries store them. That store is the KV cache [6][7]. Its size per token is fixed by the architecture: layers × KV heads × head dimension × 2 (keys and values) × bytes per number.

For the two models used here, the difference is large. Qwen2.5-0.5B has 24 layers, 2 KV heads and 64 dimensions per head [7][8], which gives 12,288 bytes per token in BF16, or 24 MiB for a 2,048-token prompt. Qwen3-0.6B has 28 layers, 8 KV heads and 128 dimensions [8][9], which gives 114,688 bytes per token and 224 MiB for the same prompt. The cache grows linearly with context and with the number of concurrent users, while the weights stay fixed. At long context, the cache is the part that runs out of memory first.

2. Existing Approaches (2025-2026 state of the art) #

flowchart TD
    K[KV cache compression] --> Q[Quantization: fewer bits per number]
    K --> E[Eviction: fewer tokens]
    K --> A[Architecture: fewer KV heads or layers]
    Q --> Q1[Per-token keys and values]
    Q --> Q2[KIVI: keys per channel, values per token]
    Q --> Q3[KVQuant and KVTuner: pre-RoPE, mixed precision]
    E --> E1[Position rules: window, sinks]
    E --> E2[Attention scores, question-agnostic: H2O]
    E --> E3[Attention scores, question-aware: SnapKV]
    E --> E4[Head-aware budgets: Ada-KV, DuoAttention]

2.1 KV cache quantization #

KIVI started from one observation: key caches have a few channels with very large magnitudes, while value caches do not. Its authors therefore quantize keys per channel (one scale per channel over a group of tokens) and values per token, which supports a tuning-free 2-bit cache [2][3]. KVQuant goes further. It quantizes keys before the rotary position embedding (RoPE) is applied, uses non-uniform data types and isolates outliers per vector, and reports less than 0.1 perplexity degradation at 3 bits on WikiText-2 and C4 [9][10]. KVTuner (ICML 2025) searches for a precision pair per layer. It finds that keys matter more than values, and that Qwen2.5-7B-Instruct needs 4.0 bits to stay nearly lossless where Llama-3.1-8B-Instruct manages with 3.25 [10][11]. A separate ICML 2025 study traced the large key values to RoPE and found that quantization schemes which ignore them lose most on tasks that rely on the context [11][12].

Work from late 2026 moves toward mixed and adaptive precision. ReadKV keeps an 8-bit cache but reads on average 4 bits per query, raising C4 perplexity by at most 0.66% across six base models [12][13]. KV4-RL uses INT4 caches for reinforcement-learning rollouts but keeps attention sinks and recent tokens at INT8. It reports a 1.80 times cumulative rollout speedup for Qwen3-8B-Base, with final accuracy within 0.5 points of BF16 rollouts [13][14].

2.2 KV cache pruning and eviction #

The simplest eviction rule keeps a sliding window of recent tokens. StreamingLLM showed that this rule breaks once the text outgrows the window, and that keeping the KV of a few initial tokens, the “attention sinks”, largely restores language modelling [14][15]. An ICLR 2025 study found such sinks in models of every size, even small ones, and describes them as acting like key biases that soak up attention without adding much to the output [15][16].

Score-based methods pick tokens using attention. H2O keeps a mix of recent tokens and “heavy hitters”, the small set of tokens that collect most of the attention [16][17]. SnapKV scores tokens by the attention they get from an observation window at the end of the prompt, which usually holds the question. It reports a 3.6 times faster generation and 8.2 times better memory efficiency at 16K tokens [17][18]. Ada-KV gives each head its own budget and tests both question-aware and question-agnostic settings [18][19]. DuoAttention keeps a full cache only for “retrieval heads” and a short window for the rest, cutting memory by up to 1.67 times for grouped-query attention (GQA) models [19][20]. Two 2026 papers question the timing and the scorer of eviction. One finds that the best scoring layer sits between 0.4 and 0.75 of the model’s depth, and that this choice beats per-head SnapKV by 18 to 43 points on five models up to 14B under tight budgets [20][21]. DeferKV delays the eviction decision from the end of prefill to the first decoding step [21][22].

2.3 How compression is evaluated #

Long-context evaluation is about more than passing a single needle. With a full cache, models already use information at the start or end of a context better than information in the middle [22][23]. LongBench (ACL 2024) covers bilingual multi-task understanding [23][24]. NoLiMa removes literal word overlap between the needle and the question; at 32K tokens, 11 of 13 models fall below half of their short-context scores [24][25]. For compression in particular, the setting matters. SCBench (ICLR 2025) follows the KV cache through its whole lifecycle with shared contexts and several queries [25][26]. KVzip reports that query-aware eviction degrades under multi-query use even at a 90% budget [26][27]. KVFundaBench finds retrieval tasks robust under aggressive compression, while reasoning tasks degrade sharply [27][28]. A 2026 serving benchmark of KIVI, SnapKV and others on Llama-3.1-8B and Mistral-7B found KIVI at 4 bits the most stable for quality, and found the compression ratio alone a poor predictor of end-to-end performance [28][29]. KVDiagnosis compares each compressed run with a full-cache control on the same input, so it can isolate the cases that compression alone breaks [29][30]. An MLSys 2025 study adds a warning on speed: current attention kernels are not built for compressed caches, so memory savings do not automatically become throughput [30][31]. A TMLR 2025 survey groups all of this into token-level, model-level and system-level methods [31][32].

Two gaps remain. Few results show quantization and eviction side by side, on the same small models, at the level of individual needle depths. And few publish runnable code that works on a CPU. We aim to fill both.

3. Method and Data #

3.1 Models and setup #

We ran Qwen2.5-0.5B-Instruct (Apache-2.0) and Qwen3-0.6B (Apache-2.0, chat template in non-thinking mode) at pinned Hugging Face revisions. The weights streamed straight into RAM with a SHA-256 check, so no weight file was written to disk. We first tried Llama-3.2-1B-Instruct, but at fp32 its 1.24 billion parameters did not fit beside the other services on the 12 GB machine, and the out-of-memory killer stopped the run. The two Qwen models make a useful pair anyway. Qwen2.5 uses a bias on its query, key and value projections [7][8]; Qwen3 removed that bias and added QK-Norm [8][9]. Their caches also differ in size by a factor of 9.33.

Weights and activations ran in fp32 because this AVX2 CPU has no native BF16 arithmetic. The baseline cache is rounded to BF16, and every byte count is relative to 16 bits per number. All runs used 6 threads, torch 2.10.0 and transformers 5.2, with seed 0. The two final runs took 694.1 seconds of wall time in total.

3.2 Needle-in-a-haystack protocol #

Each prompt is a chat turn. It opens with an instruction (“There is a pass key hidden inside a lot of irrelevant text…”). Next comes filler text built from HellaSwag contexts [32][33], with the needle “The pass key is NNNNN. Remember it. NNNNN is the pass key.” placed at a sentence boundary at 5%, 25%, 50%, 75% or 95% depth. The question, “What is the pass key?”, comes last, and the assistant’s reply begins “The pass key is”. Prompts are 1,024 or 2,048 tokens long. Qwen2.5 received 2 random passkeys per (length, depth) cell, 20 prompts in total. Qwen3 received 1 per cell, 10 prompts, to fit the compute budget.

The order of operations matters, so we fixed it. The context (everything before the question) is prefilled once in full precision. Then the cache is compressed, and only after that are the question and answer fed in. We call this compress-then-ask. It is the situation of a cached document, a system prompt reused across turns, or a long conversation history: compression happens before the next question is known. The question and answer tokens themselves stay uncompressed, much like KIVI’s full-precision window of recent tokens.

The five digits of the answer (6 tokens including the leading space) are teacher-forced. A prompt counts as correct only if the model’s top prediction matches every answer token, which is exactly the condition under which greedy decoding prints the right passkey. We also record the answer’s log-probability.

flowchart LR
    P[Chat prefix plus haystack with needle] --> F[Full-precision prefill]
    F --> C{Compress context cache}
    C --> Q1[Quantize: per-token or KIVI layout]
    C --> Q2[Evict: window, sinks, H2O, SnapKV]
    Q1 --> A[Feed question plus teacher-forced answer]
    Q2 --> A
    A --> M[Exact match, log-probability, needle tokens kept]

3.3 The 15 cache treatments #

MethodRuleContext KV bytes vs BF16
bf16Rounded to bfloat16 (baseline)100%
tok-int8 / int4 / int2Asymmetric min-max, keys and values per token, groups of 32 channels56.25% / 31.25% / 18.75%
kivi-int8 / int4 / int2Keys per channel (groups of 32 tokens), values per token56.25% / 31.25% / 18.75%
window-50 / 25Keep the most recent 50% or 25% of context tokens50% / 25%
sink-50 / 254 first tokens plus the most recent tokens50% / 25%
h2o-50 / 25Half recent, half top tokens by attention summed over all context queries, per KV head50% / 25%
snapkv-50 / 25Top tokens by attention from the question tokens, max-pooled with kernel 7, per KV head50% / 25%

Byte fractions are computed: INT-b costs b bits plus a 16-bit scale and a 16-bit zero point per group of 32 numbers, which adds 1 bit per number. Source: our setup, results.json.

Quantization is simulated: each number is quantized and immediately dequantized, so the model sees exactly the rounding error a real INT kernel would produce, but nothing runs faster. Keys are quantized after RoPE, as stored by the transformers cache. Our H2O is a one-shot version applied at the end of prefill and does not see the question. Our SnapKV does see the question, which is its best case, and it would have to recompress for every new question. On the WikiText-2 perplexity test there is no question, so SnapKV used the last 32 prefix tokens as its observation window, as in the original paper.

3.4 Perplexity, latency and checks #

For perplexity, the first 1,024 tokens of the WikiText-2 test set [33][34] are cached and compressed, and the negative log-likelihood (NLL) of the next 256 tokens is measured. As a reference, we also scored the same 256 tokens with no context at all. Decode latency is the median of 16 greedy single-token steps after a 2,048-token prompt, repeated 3 times, with the full cache and with 50% and 25% windows. Two checks guard the code. Cached and single-pass forward logits differ by at most 7e-05 (Qwen2.5) and 5e-05 (Qwen3). The eviction code with every token kept reproduces the baseline logits exactly, a difference of 0.0.

Code, data, every prompt’s raw output and the charts: stabilarity/hub/research/kv-cache-quantization-benchmark.

4. Results #

4.1 RQ1, KV cache quantization benchmark: INT8 vs INT4 vs INT2 #

Two heatmaps of passkey retrieval accuracy for 15 KV cache methods across five needle depths at 1,024 and 2,048 tokens, for Qwen2.5-0.5B-Instruct and Qwen3-0.6B; bf16, INT8, KIVI INT4 and SnapKV rows are fully correct, the per-token INT2 row is fully wrong, and window and sink rows fail at shallow depths

At 8 bits, both layouts behave as if nothing happened. Every INT8 variant retrieved every passkey, and perplexity moved by at most 0.25%. Per-token INT8 gave a ratio of 1.0025 on Qwen2.5, while KIVI INT8 reproduced the baseline NLL to five decimals on Qwen2.5 (1.80696) and gave 0.9991 of baseline perplexity on Qwen3.

At 4 bits, the layout decides the outcome. KIVI INT4 retrieved 20 of 20 passkeys on Qwen2.5 and 10 of 10 on Qwen3, with perplexity ratios of 1.013 and 1.0101. Per-token INT4 retrieved 10 of 20 on Qwen2.5 (50%, 95% interval 29.93-70.07%) and raised perplexity by 36.48% on Qwen2.5 and 43.9% on Qwen3. It still found every passkey on Qwen3 (10 of 10). Its Qwen2.5 failures are telling: the model did not lose the needle, it misread it. Under teacher forcing it produced 77713 for 77013, 86255 for 86465 and 76152 for 76150. The digits are too blurred to read, but the needle’s location survives, and the mean answer log-probability fell from -0.0506 to -3.2463.

At 2 bits, per-token quantization destroys the cache. It retrieved 0 of 30 passkeys across both models. Perplexity rose to 100.4088 on Qwen2.5 and 19,364.7765 on Qwen3, both far worse than the 37.5226 and 53.3163 the models reach on the same tokens with no context at all. A broken cache is worse than an empty one. KIVI INT2 kept 18 of 20 on Qwen2.5 and 9 of 10 on Qwen3, with perplexity ratios of 1.2484 and 1.0955. Both Qwen2.5 misses were at 50% depth in a 1,024-token prompt. The single Qwen3 miss was a formatting slip, not a retrieval failure: at 2,048 tokens and 50% depth, the model’s top first token was “**” (Markdown bold), and the digits that followed were correct.

Left: bar chart of per-channel maximum absolute key and value activations in one Qwen2.5-0.5B layer, with key channels reaching about 11 against values below about 6; right: mean relative quantization error for keys and values quantized per token or per channel at INT4 and INT2 for both models

The reason sits in the keys. Averaged over layers, Qwen2.5’s largest key channel is 5.24429 times its median channel, against 2.1015 for values. In Qwen3 the ratio is 15.09986 for keys and 2.78339 for values, and one Qwen3 layer reaches 74.551 (per-layer data in raw/qwen3-0.6b.json), so QK-Norm does not remove the outliers. That matches the finding that RoPE concentrates large values in keys but not in values [11][12]. When a whole token shares one scale, a single large channel stretches the range and wipes out the small channels. Per-channel grouping keeps each channel’s range separate. The measured reconstruction error shows this directly. At INT4, the relative key error drops from 0.09602 per token to 0.03144 per channel on Qwen2.5, and from 0.10671 to 0.03305 on Qwen3. At INT2, it drops from 0.48485 to 0.1587 and from 0.50476 to 0.16655. For values, the per-channel layout helps far less (0.08737 to 0.06672 at INT4 on Qwen2.5), which is why KIVI keeps values per token.

MethodKV bytes vs BF16Qwen2.5 NIAHQwen3 NIAHQwen2.5 PPL ratioQwen3 PPL ratioQwen3 context KV at 2k
bf16100%20/2010/101.01.0224.0 MiB
tok-int856.25%20/2010/101.00250.999126.0 MiB
kivi-int856.25%20/2010/101.00.9991126.0 MiB
tok-int431.25%10/2010/101.36481.43970.0 MiB
kivi-int431.25%20/2010/101.0131.010170.0 MiB
tok-int218.75%0/200/1016.48232,845.668842.0 MiB
kivi-int218.75%18/209/101.24841.095542.0 MiB
window-5050%11/206/1073.1459303.575112.0 MiB
sink-5050%12/206/103.21653.6854112.0 MiB
h2o-5050%12/209/101.96981.4762112.0 MiB
snapkv-5050%20/2010/101.3891.1883112.0 MiB
window-2525%4/202/1080.1126451.053956.0 MiB
sink-2525%7/202/103.49264.128656.0 MiB
h2o-2525%4/207/102.83552.211156.0 MiB
snapkv-2525%20/2010/102.47021.897656.0 MiB

NIAH = passkey prompts retrieved exactly (1k and 2k, five depths). PPL ratio = WikiText-2 perplexity on 256 tokens after a compressed 1,024-token prefix, divided by the BF16-cache perplexity (6.0919 for Qwen2.5, 6.805 for Qwen3). KV at 2k = computed size of a 2,048-token context cache. Source: our measurements, results.json.

4.2 RQ2, KV cache pruning: window, sinks and heavy hitters #

Log-scale bar chart of WikiText-2 perplexity relative to the BF16 cache for 15 KV cache methods on two models; sliding windows reach 73 to 451 times baseline, sink and H2O methods 1.5 to 4.1 times, KIVI INT4 about 1.01 times, and per-token INT2 16.5 and 2,846 times

A sliding window is the worst way to halve a cache. On text, dropping the first tokens makes perplexity explode: a 50% window raised it 73.1459 times on Qwen2.5 and 303.575 times on Qwen3, worse than having no context at all. Keeping just 4 sink tokens on top of the same window brings the ratios down to 3.2165 and 3.6854. That is the StreamingLLM effect [14][15], and it is present at 0.5B parameters, as the attention-sink study predicts [15][16].

Sinks fix fluency, though, not retrieval. On the needle test, the window and sink methods behave almost the same, because both keep only the most recent tokens. At a 50% budget, both failed at 5% and 25% depth in every prompt for both models and succeeded at 75% and 95%. At 25%, only the needles at 95% depth were retrieved every time. Sink-25 retrieved 7 of 20 on Qwen2.5 and 2 of 10 on Qwen3. Window-25 retrieved 4 of 20 and 2 of 10.

H2O sits in between, and its behaviour depends on the model. On Qwen3, h2o-50 retrieved 9 of 10 and h2o-25 7 of 10. On Qwen2.5, they retrieved 12 of 20 and 4 of 20. Accumulated attention is a question-agnostic signal: it keeps tokens that the context itself found interesting. A passkey sentence attracts some of that attention, but not reliably, and on Qwen2.5 the h2o-50 cache missed the needle at 50% depth in all 4 prompts.

SnapKV never missed. It retrieved 30 of 30 at a 50% budget and 30 of 30 at 25%. That is expected: it chose tokens after seeing the question, and the question points almost directly at the needle. It is also its best case. In our compress-then-ask setting, a real deployment would have to keep the full cache until each question arrives, or recompress per question. KVzip’s multi-query results show that query-aware selection fitted to one question degrades on the next [26][27].

4.3 Needle in a haystack: where compressed caches fail #

Two scatter plots of passkey retrieval accuracy against the percentage of needle tokens still in the cache for eviction methods on Qwen2.5-0.5B and Qwen3-0.6B; SnapKV keeps about 81 to 90 percent of needle tokens and retrieves 100 percent, windows and sinks sit near the diagonal, and Qwen3 H2O sits above it

For eviction, retrieval follows retention. We counted how many of the needle’s 26 tokens (the same count on both models) were still in the cache, averaged over layers and KV heads. Windows and sinks sit close to the line where accuracy equals retention. Window-50 kept 60% of needle tokens on both models and retrieved 55% (Qwen2.5) and 60% (Qwen3). Window-25 and sink-25 kept between 25% and 33.85% of needle tokens and retrieved between 20% and 35% of passkeys. SnapKV kept 81.88% of needle tokens at a 25% budget on Qwen2.5 and 80.85% on Qwen3, more than three times the share of the budget, and retrieved everything.

H2O on Qwen3 sits clearly above the line. With 33.28% of needle tokens kept on average, h2o-25 still retrieved 7 of 10. Because the selection is made per KV head, a passkey that survives in some heads and layers can be enough. On Qwen2.5, with only 2 KV heads, the same rule kept 31.14% and retrieved 4 of 20. Fewer heads give fewer chances for a stray needle to survive.

Depth matters for every eviction rule except SnapKV, but in the opposite direction from the “lost in the middle” effect [22][23]. With the full BF16 cache, both models retrieved the passkey at every depth and both lengths. Compression created the depth effect: recency-based rules lose early needles, and the heavy-hitter rule loses needles that the surrounding text did not attend to. For quantization, depth mattered little. KIVI INT2’s three misses were all at 50% depth, but with 1 or 2 prompts per cell, that pattern is weak evidence.

4.4 RQ3, which compression to use: accuracy per byte and latency #

Two scatter plots of passkey retrieval accuracy against KV cache size as a percentage of BF16 bytes for 15 methods on Qwen2.5-0.5B and Qwen3-0.6B; KIVI INT4 at 31.25 percent and SnapKV at 25 percent reach 100 percent, KIVI INT2 at 18.75 percent reaches 90 percent, per-token INT2 reaches 0 percent

Pooled over both models (30 prompts per method), the methods that keep full accuracy are BF16, both INT8 layouts, KIVI INT4, and SnapKV at 50% and 25%. Their 95% interval is 88.65-100%. Among the question-agnostic methods, KIVI INT4 is the cheapest full-accuracy option, at 31.25% of BF16 bytes. KIVI INT2 at 18.75% retrieved 27 of 30 (90%, interval 74.38-96.54%). The best question-agnostic eviction at a 50% budget, H2O, retrieved 21 of 30 (70%, interval 52.12-83.34%) while using more bytes than KIVI INT2. For Qwen3, KIVI INT4 shrinks a 2,048-token context cache from 224 MiB to 70 MiB with no measured loss in retrieval and a 1.01% rise in perplexity.

Eviction does buy real speed, which our simulated quantization cannot. Decoding after a 2,048-token prompt took a median 185.3 ms per token with the full Qwen2.5 cache (2,026 context tokens) and 170.47 ms with a 25% window (506 tokens). That is 8% less, by our arithmetic. For Qwen3, the time fell from 252.54 ms to 222.56 ms, or 11.9% less. The gain is small because at this context length a small model’s decode step is dominated by reading the weights, not the cache. The steps ran in fp32 on a shared server, so only the relative change is meaningful. Quantized caches need dedicated kernels to turn bytes into speed. Transformers ships a quantized cache with int2 to int8 backends [6][7], but the MLSys 2025 analysis warns that such savings do not automatically raise serving throughput [30][31].

flowchart TD
    S[Need a smaller KV cache] --> Q{Is the question known before compression?}
    Q -->|No: cached documents, multi-turn chats| A{Memory target?}
    Q -->|Yes: single question per context| B[Question-aware eviction such as SnapKV, 25 to 50 percent budget]
    A -->|About half| C[INT8, any layout: no measured loss]
    A -->|About a third| D[KIVI INT4: keys per channel, values per token]
    A -->|Under a fifth| E[KIVI INT2: expect occasional misses]
    A -->|Need decode speed too| F[Sinks plus heavy hitters, never a plain window]
    D --> V[Validate on your own retrieval and reasoning tasks]
    E --> V
    F --> V
    B --> V

5. Discussion #

Which KV cache compression to use. If the compressed cache must serve questions that are not yet known, quantize before you evict. In our runs, an INT8 cache was indistinguishable from BF16, and a 4-bit cache in the KIVI layout kept every needle on both models at under a third of the bytes. Both the original KIVI paper [2][3] and the 2026 serving benchmark that found KIVI at 4 bits the most stable method [28][29] point the same way. The layout matters more than the bit width: per-token INT4 lost half the Qwen2.5 needles while KIVI INT2, with fewer bits, lost 2 of 20. When choosing a library, check how it groups keys, not just how many bits it uses. If you must evict, never drop the first tokens. A plain window raised perplexity 73 to 451 times, while 4 sink tokens kept it under 4.2 times. A question-aware selector is excellent when there is one question per context and risky when the cache is reused.

Why the two models differ. Qwen3’s larger cache (8 KV heads of 128 dimensions) tolerated per-token INT4 and H2O eviction better than Qwen2.5’s 2 KV heads of 64 dimensions. More heads give the needle more independent places to survive. For the cost-minded reader of this series, the trade-off is direct: Qwen3 needs 9.33 times more cache per token, but it holds up better when that cache is compressed hard. KIVI INT4 brings Qwen3’s 2,048-token context to 70 MiB, which is still almost three times Qwen2.5’s uncompressed 24 MiB.

Limitations and threats to validity.

  • Sample size. The tests used 20 and 10 prompts per method per model, with 1 or 2 per cell. A 10 of 10 result has a 95% interval of 72.25-100%. Rankings among methods at or near 100% are not established.
  • Short contexts, small models. The prompts were 1k and 2k tokens on 0.5-0.6B models. At 32k and beyond, or on 7B+ models, eviction budgets bite harder and quantization error accumulates over more tokens. RULER and NoLiMa show that difficulty rises with length even without compression [5][6] [24][25].
  • Easy task. A passkey with literal word overlap is retrieval, the task that KVFundaBench found most robust to compression. Reasoning degrades sooner [27][28].
  • Simulated quantization. We modelled the rounding exactly, but not real kernels. The byte counts are computed. Keys were quantized after RoPE; pre-RoPE quantization, as in KVQuant, should do better [9][10].
  • Simplified baselines. Our H2O is one-shot and our SnapKV sees the question. Per-head budgets [18][19], a mid-depth scoring layer [20][21] or deferred eviction [21][22] would change the eviction numbers.
  • Uncompressed question. The question and answer stayed at full precision. During long generation, newly generated tokens would also be compressed.

What would change the conclusion. Three results would. A long-context run in which KIVI INT4 loses needles that INT8 keeps would cap the safe level at 8 bits. A question-agnostic eviction method that matches KIVI INT4’s accuracy at a 25% budget on these models would make eviction the cheaper route, since it also cuts attention compute. And a real INT4 kernel that is slower than BF16 on CPU would make the memory saving the only benefit.

6. Conclusion #

RQ1 Finding: A 4-bit KV cache does not lose the needle if keys are quantized per channel. Measured by passkey retrieval pooled over two models = 30 of 30 for KIVI-layout INT4 at 31.25% of BF16 bytes, against 20 of 30 for per-token INT4 and 0 of 30 for per-token INT2. KIVI INT2 retrieved 27 of 30 at 18.75%, and its perplexity ratio stayed at 1.2484 or lower. This matters for the series because KIVI INT4 would cut Qwen3-0.6B’s 2,048-token context cache from 224 MiB to 70 MiB with a 1.01% rise in perplexity.

RQ2 Finding: KV cache pruning fails by position, not by budget. Measured by passkey retrieval at a 25% budget = 6 of 30 for a sliding window, 9 of 30 with attention sinks, 11 of 30 for one-shot H2O and 30 of 30 for question-aware SnapKV. Every recency-based rule lost needles placed at 5% and 25% depth. This matters for the series because a sliding window without sinks also raised perplexity 73 to 451 times, so a cheap eviction rule can silently break a deployment.

RQ3 Finding: Per byte, quantization in the KIVI layout gives the most question-agnostic accuracy, and eviction gives the only real speed gain on a CPU. Measured by retrieval per byte = 100% at 31.25% (KIVI INT4) and 90% at 18.75% (KIVI INT2), against 70% at 50% for the best question-agnostic eviction, H2O. Measured by decode time with a 25% window = 8% lower for Qwen2.5 and 11.9% lower for Qwen3. This matters for the series because on small models memory, not decode time, is what compression saves.

The next question is whether these results hold at 32k tokens and on reasoning tasks, where the cache is larger and the needle harder to find.

References (34) #

  1. Stabilarity Research Hub. (2026). KV Cache Quantization Benchmark: 4-Bit KIVI Finds 30 of 30 Needles. doi.org. dtl
  2. Stabilarity Research Hub. (2026). Llama vs Qwen vs Gemma on a CPU: 3.5 GiB of KV Cache for a 0.6B Model. tb
  3. Liu, Zirui, Yuan, Jiayi, Jin, Hongye, Zhong, Shaochen, et al.. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. doi.org. dtil
  4. gkamradt. (2023). gkamradt/LLMTest_NeedleInAHaystack (GitHub repository). github.com. tr
  5. Stabilarity Research Hub. (2026). Long-Context Retrieval Benchmarks — Needle-in-Haystack and Beyond. tb
  6. (2024). [2404.06654] RULER: What's the Real Context Size of Your Long-Context Language Models?. doi.org. dtil
  7. (2026). huggingface.co. r
  8. (2024). [2412.15115] Qwen2.5 Technical Report. doi.org. dtil
  9. (2025). [2505.09388] Qwen3 Technical Report. doi.org. dtil
  10. Hooper, Coleman, Kim, Sehoon, Mohammadzadeh, Hiva, Mahoney, Michael W., et al.. (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. doi.org. dtil
  11. (2025). [2502.04420] KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference. doi.org. dtil
  12. Jin, Mingyu, Mei, Kai, Xu, Wujiang, Sun, Mingjie, et al.. (2025). Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding. doi.org. dtil
  13. Bhandari, Siddharth, Gretta, Lucas, Balasubramanian, Krishna, Kasiviswanathan, Shiva. (2026). Read What Matters: Query-Adaptive Quantization for KV Caches. doi.org. dtil
  14. Park, Sihwan, Kim, Sung-Yub, Jang, Doohyuk, Yang, Eunho. (2026). KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization. doi.org. dtil
  15. Xiao, Guangxuan, Tian, Yuandong, Chen, Beidi, Han, Song, et al.. (2023). Efficient Streaming Language Models with Attention Sinks. doi.org. dtil
  16. Gu, Xiangming, Pang, Tianyu, Du, Chao, Liu, Qian, et al.. (2025). When Attention Sink Emerges in Language Models: An Empirical View. doi.org. dtil
  17. O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models" data-ref-authors="Zhang, Zhenyu, Sheng, Ying, Zhou, Tianyi, Chen, Tianlong, et al." data-ref-year="2023" data-ref-source="doi.org" data-ref-url="https://doi.org/10.48550/arXiv.2306.14048" data-ref-accessed="Mar 28, 2026" data-ref-dbid="10262" data-ref-type="doi" data-crossref="0" data-doi="1" data-peer="0" data-trusted="1" data-indexed="1" data-access="free">Zhang, Zhenyu, Sheng, Ying, Zhou, Tianyi, Chen, Tianlong, et al.. (2023). HO: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. doi.org. dtil
  18. Li, Yuhong, Huang, Yingbing, Yang, Bowen, Venkitesh, Bharat, et al.. (2024). SnapKV: LLM Knows What You are Looking for Before Generation. doi.org. dtil
  19. Feng, Yuan, Lv, Junlin, Cao, Yukun, Xie, Xike, et al.. (2025). Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. doi.org. dtil
  20. Xiao, Guangxuan, Tang, Jiaming, Zuo, Jingwei, Guo, Junxian, et al.. (2024). DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. doi.org. dtil
  21. Seregina, Ekaterina. (2026). Who Should Score the Cache? Mid-Depth Layers Are the Best Judges of KV-Cache Eviction. doi.org. dtil
  22. Wang, Zhe, Li, Jiakai, Sun, Yujia, Wang, Rongzheng, et al.. (2026). DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression. doi.org. dtil
  23. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, et al.. (2024). Lost in the Middle: How Language Models Use Long Contexts. doi.org. dcrtil
  24. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, et al.. (2024). LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. doi.org. dcrtil
  25. Modarressi, Ali, Deilamsalehy, Hanieh, Dernoncourt, Franck, Bui, Trung, et al.. (2025). NoLiMa: Long-Context Evaluation Beyond Literal Matching. doi.org. dtil
  26. Li, Yucheng, Jiang, Huiqiang, Wu, Qianhui, Luo, Xufang, et al.. (2025). SCBench: A KV Cache-Centric Analysis of Long-Context Methods. doi.org. dtil
  27. Kim, Jang-Hyun, Kim, Jinuk, Kwon, Sangwoo, Lee, Jae W., et al.. (2025). KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction. doi.org. dtil
  28. Liu, Xiang, Tang, Zhenheng, Chen, Hong, Dong, Peijie, et al.. (2025). Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression. doi.org. dtil
  29. Agrawal, Nikita, Mayer, Ruben. (2026). Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving. doi.org. dtil
  30. Qiu, Chen, Liu, Ziwu, Fei, Chao, Li, Guozhong, et al.. (2026). KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models. doi.org. dtil
  31. Gao, Wei, Zhou, Xinyu, Sun, Peng, Zhang, Tianwei, et al.. (2025). Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving. doi.org. dtil
  32. (2025). [2412.19442] A Survey on Large Language Model Acceleration based on KV Cache Management. doi.org. dtil
  33. Zellers, Rowan; Holtzman, Ari; Bisk, Yonatan; Farhadi, Ali; Choi, Yejin. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?. doi.org. dcta
  34. (2026). huggingface.co. r
← Previous
Llama vs Qwen vs Gemma on a CPU: 3.5 GiB of KV Cache for a 0.6B Model
Next →
Next article coming soon
All Cost-Effective Enterprise AI articles (45)45 / 45
Version History · 2 revisions
+
RevDateStatusActionBySize
v1Nov 17, 2026DRAFTInitial draft
First version created
(w) Author0 (~0)
v2Nov 17, 2026CURRENTPublished
Article published to research hub
(w) Author29,409 (+29409)

Versioning is automatic. Each revision reflects editorial updates, reference validation, or formatting changes.

Recent Posts

  • AI Data Readiness Score: Label Noise Hurts 5x More Than Missing Data
  • KV Cache Quantization Benchmark: 4-Bit KIVI Finds 30 of 30 Needles
  • 7 Open Source Humanoid Robots Pushed in Simulation: Feet Beat Torque
  • Llama vs Qwen vs Gemma on a CPU: 3.5 GiB of KV Cache for a 0.6B Model
  • How Long Can a Humanoid Robot Hold a Box? Joint Torque, Motor Heat and Battery Limits on Five Open Robot Models

Research Index

Browse all articles — filter by score, badges, views, series →

Categories

  • ai
  • AI Economics
  • AI Memory
  • AI Observability & Monitoring
  • AI Portfolio Optimisation
  • Ancient IT History
  • Anticipatory Intelligence
  • Article Quality Science
  • Capability-Adoption Gap
  • Cost-Effective Enterprise AI
  • Future of AI
  • Geopolitical Risk Intelligence
  • hackathon
  • healthcare
  • HPF-P Framework
  • innovation
  • Intellectual Data Analysis
  • Living NPCs
  • medai
  • Medical ML Diagnosis
  • Open Humanoid
  • Research
  • ScanLab
  • Shadow Economy Dynamics
  • Spec-Driven AI Development
  • Technology
  • Trusted Open Source
  • Uncategorized
  • Universal Intelligence Benchmark
  • War Prediction
  • Кафедра ЕКІТ

About

Stabilarity Research Hub is dedicated to advancing the frontiers of AI, from Medical ML to Anticipatory Intelligence. Our mission is to build robust and efficient AI systems for a safer future.

Language

  • Medical ML Diagnosis
  • AI Economics
  • Cost-Effective AI
  • Anticipatory Intelligence
  • Data Mining
  • 🔑 API for Researchers

Connect

Facebook Group: Join

Telegram: @Y0man

Email: contact@stabilarity.com

© 2026 Stabilarity Research Hub

© 2026 Stabilarity Hub | Powered by Superbs Personal Blog theme
Stabilarity Research Hub

Open research platform for AI, machine learning, and enterprise technology. All articles are preprints with DOI registration via Zenodo.

410+
Articles
20+
Series
DOI
Archived

Research Series

  • Medical ML Diagnosis
  • Cost-Effective Enterprise AI
  • Future of AI
  • Trusted Open Source
  • Geopolitical Risk Intelligence
  • Capability–Adoption Gap
  • Spec-Driven AI
  • Shadow Economy Dynamics

Community

  • EKIT Department
  • Join Community
  • MedAI Hack
  • Zenodo Collection
  • GitHub
  • contact@stabilarity.com

Legal

  • Terms of Service
  • About Us
  • Contact
  • CC BY 4.0 License
Operated by
Stabilarity OÜ
Registry: 17150040
Estonian Business Register →
© 2026 Stabilarity OÜ. Content licensed under CC BY 4.0
Terms About Contact
Language: 🇬🇧 EN 🇺🇦 UK 🇩🇪 DE 🇵🇱 PL 🇫🇷 FR
Display Settings
Theme
Light
Dark
Auto
Width
Default
Column
Wide
Text 100%

We use cookies to enhance your experience and analyze site traffic. By clicking "Accept All", you consent to our use of cookies. Read our Terms of Service for more information.