---

Abstract #

The KV-cache memory bottleneck in large language model inference has generated three competing families of compression techniques — quantization, token eviction, and structured pruning — each claiming substantial memory savings with minimal accuracy loss. This article benchmarks these approaches head-to-head, drawing on 2026 research that provides standardized comparisons across architectures and tasks. We formulate three research questions addressing compression ratio versus accuracy trade-offs, the interaction between compression strategy and task type, and hybrid approaches that combine multiple techniques. Our analysis reveals that quantization methods achieve the most consistent accuracy preservation at moderate compression (2-4x), eviction strategies excel at extreme compression ratios (8x+) but introduce irreversible information loss, and emerging hybrid frameworks like ARKV and KVTC that combine quantization with adaptive eviction achieve the best Pareto-optimal trade-offs — preserving approximately 97% of baseline accuracy while reducing memory by 4x. These findings establish the empirical foundation for selecting compression strategies based on deployment constraints, directly informing the optimization techniques explored in subsequent articles of the AI Memory series.

1. Introduction #

In the previous article, we established that model accuracy degrades following non-linear sigmoid decline functions as context length increases, with effective context utilization plateauing at 60-70% of advertised capacity (Ivchenko, 2026^[2]). These degradation curves are not merely theoretical — they define the operational envelope within which compression techniques must function. If a model already loses 13.9-85% of its performance at full context length, any compression strategy that introduces additional accuracy loss must be evaluated against an already-degraded baseline.

The KV-cache grows linearly with sequence length and batch size, quickly dominating GPU memory during inference. For a 70B-parameter model processing 128K tokens, the cache alone can consume over 40GB of HBM — often exceeding the memory footprint of the model weights themselves (Dong et al., 2026^[3]). This memory pressure creates a fundamental deployment constraint: either limit context length (sacrificing capability), reduce batch size (sacrificing throughput), or compress the cache (risking accuracy).

Three families of compression techniques have emerged to address this constraint. Quantization reduces the numerical precision of cached key-value states, typically from FP16 to INT8 or lower. Eviction selectively discards tokens deemed less important based on attention patterns or heuristic scoring. Pruning removes structured portions of the cache — entire heads, layers, or channel dimensions — based on redundancy analysis. Each approach makes fundamentally different trade-offs between compression ratio, accuracy preservation, computational overhead, and reversibility.

Research Questions #

RQ1: What are the empirical compression-accuracy trade-off curves for quantization, eviction, and pruning when benchmarked under standardized conditions across multiple model architectures?

RQ2: How does the choice of compression strategy interact with downstream task type — specifically, do different tasks (long-context retrieval, reasoning, summarization) favor fundamentally different compression families?

RQ3: Can hybrid approaches that combine multiple compression techniques achieve Pareto-superior trade-offs compared to single-strategy methods, and what are the practical engineering constraints of such combinations?

2. Existing Approaches (2026 State of the Art) #

2.1 Quantization-Based Compression #

Quantization reduces the bit-width of cached key and value tensors. The simplest approach — uniform INT8 quantization — achieves a straightforward 2x compression by halving the bytes per element. Dong et al. (2026) implemented GPU-accelerated INT8 quantization kernels achieving 4x memory reduction with reconstruction error below 0.004 and attention score error below 0.1 (Dong et al., 2026^[3]). Their vectorized CUDA kernel achieves up to 1,694x speedup over CPU baselines, demonstrating that quantization’s computational overhead is negligible at 6-58ms per operation.

More aggressive approaches push to 2-bit quantization. KVTuner introduces sensitivity-aware layer-wise mixed-precision quantization, allocating different bit-widths to different layers based on their sensitivity to quantization error (Wang et al., 2026^[4]). This achieves 21.25% inference throughput improvement by concentrating compression on less-sensitive layers while preserving full precision where it matters most.

VQKV takes a fundamentally different approach by applying vector quantization — representing groups of floating-point values as indices into a learned codebook. On LLaMA3.1-8B, VQKV achieves 82.8% compression ratio while retaining 98.6% of baseline performance on LongBench, enabling 4.3x longer generation on the same memory footprint (Chen et al., 2026^[5]). Unlike scalar quantization, vector quantization can capture inter-element correlations, yielding higher fidelity at equivalent compression ratios.

2.2 Eviction-Based Compression #

Token eviction strategies achieve compression by selectively discarding KV-cache entries for tokens deemed unimportant. The canonical approach uses cumulative attention scores as an importance proxy — tokens that have received little attention historically are evicted first. H2O (Heavy-Hitter Oracle) and SnapKV represent this family, achieving high compression ratios by retaining only the most-attended tokens (Liu et al., 2025^[6]).

KVCrush extends eviction by exploiting similarity in attention head behavior. Rather than evicting individual tokens, it identifies groups of tokens with similar key representations and replaces them with a single representative, reducing redundancy without purely discarding information (Joshi et al., 2026^[7]). This approach preserves more semantic content than simple attention-score-based eviction.

The critical limitation of eviction is irreversibility. Once a token is evicted, its information is permanently lost. Recent work by the KV Policy team reframes eviction as a reinforcement learning problem, training lightweight per-head agents to predict tokens’ future utility rather than relying on past attention as a proxy (Jegou and Jeblick, 2026^[8]). Evaluated on the RULER long-context benchmark and OASST2-4k multi-turn dialogue, KVP significantly outperforms heuristic baselines, demonstrating that learned eviction policies can substantially improve the accuracy-compression trade-off.

2.3 Pruning and Transform Coding #

Pruning removes structured components — channels, heads, or layers — from the KV-cache based on measured redundancy. Unlike token-level eviction, structural pruning maintains the sequence-length dimension but reduces the per-token memory footprint.

KVTC (KV Transform Coding) draws on classical media compression techniques, combining PCA-based feature decorrelation, adaptive quantization, and entropy coding to compress KV-caches for compact storage. On Llama 3, Mistral NeMo, and R1-Qwen 2.5 across benchmarks including AIME25, GSM8K, and RULER, KVTC achieves up to 20x compression while maintaining reasoning and long-context accuracy — and 40x or higher for specific use cases (Li et al., 2026^[9]). It consistently outperforms token eviction, scalar quantization, and SVD-based methods at equivalent compression ratios.

KVzap introduces an input-adaptive approximation that works during both prefilling and decoding phases. On Qwen3-8B, Llama-3.1-8B-Instruct, and Qwen3-32B across long-context and reasoning tasks, KVzap achieves 2-4x compression with negligible accuracy loss and achieves state-of-the-art performance on the KVpress leaderboard (Jegou et al., 2026^[10]).

flowchart TD
    A[KV-Cache Compression] --> Q[Quantization]
    A --> E[Eviction]
    A --> P[Pruning / Transform]
    Q --> Q1[INT8 Uniform: 2-4x, low risk]
    Q --> Q2[Mixed-Precision: 2-8x, moderate risk]
    Q --> Q3[Vector Quantization: 4-5x, low risk]
    E --> E1[Attention-Score: 4-10x, high risk]
    E --> E2[Learned RL Policy: 4-8x, moderate risk]
    E --> E3[Similarity Merging: 2-4x, low risk]
    P --> P1[SVD Low-Rank: 2-4x, moderate risk]
    P --> P2[Transform Coding: 10-40x, low risk]
    P --> P3[Adaptive Pruning: 2-4x, low risk]

3. Quality Metrics and Evaluation Framework #

Evaluating KV-cache compression requires metrics that capture multiple dimensions of the trade-off space. No single number adequately characterizes a compression method — a technique that excels on summarization may catastrophically fail on multi-hop reasoning.

3.1 Compression Metrics #

Compression Ratio (CR): The ratio of original cache size to compressed cache size. While straightforward, CR alone is insufficient because methods with identical CRs can have vastly different accuracy profiles.

Memory Reduction Factor (MRF): The multiplicative factor of memory savings, accounting for metadata overhead. A method claiming 4x compression but requiring 30% additional metadata for index storage has an effective MRF of only 2.9x.

Throughput Impact (TI): The change in tokens-per-second during inference. Some compression methods (especially learned eviction) introduce computational overhead that partially offsets memory savings.

3.2 Accuracy Metrics #

Baseline Retention Rate (BRR): The percentage of full-cache accuracy preserved after compression, measured as scorecompressed / scorefull_cache. This is the primary quality metric.

Task-Specific Degradation (TSD): Accuracy loss measured independently across task categories — retrieval (RULER), reasoning (GSM8K, AIME), generation (LongBench), and dialogue (OASST2).

Degradation Onset Point (DOP): The compression ratio at which accuracy drops below 95% of baseline. This identifies the practical operating limit for each method.

RQ	Metric	Source	Threshold
RQ1	Baseline Retention Rate at matched CR	ARKV, VQKV, KVzap benchmarks	BRR >= 95%
RQ2	Task-Specific Degradation variance	RULER, GSM8K, LongBench cross-task	TSD std < 5%
RQ3	Pareto frontier area vs single methods	ARKV hybrid vs components	>= 10% improvement

graph LR
    subgraph Compression_Metrics
        CR[Compression Ratio] --> MRF[Memory Reduction Factor]
        CR --> TI[Throughput Impact]
    end
    subgraph Accuracy_Metrics
        BRR[Baseline Retention] --> TSD[Task-Specific Degradation]
        BRR --> DOP[Degradation Onset Point]
    end
    CR --> BRR
    MRF --> Pareto[Pareto Frontier]
    BRR --> Pareto

4. Application to AI Memory #

4.1 Benchmarking Under Standardized Conditions (RQ1) #

Synthesizing results across the 2026 literature reveals clear compression-accuracy frontiers for each technique family. At the moderate compression regime (2-4x), quantization methods dominate. INT8 uniform quantization achieves BRR above 99% at 2x compression (Dong et al., 2026^[3]). Mixed-precision approaches like KVTuner push this to 4x while maintaining BRR above 97% by concentrating precision where sensitivity is highest (Wang et al., 2026^[4]). Vector quantization achieves a compelling middle ground — VQKV reaches 82.8% compression (approximately 5.8x) with 98.6% BRR on LongBench (Chen et al., 2026^[5]).

At extreme compression (8x+), eviction becomes the only viable single-strategy approach, but at significant accuracy cost. Attention-score eviction at 80% token removal (5x effective compression) can maintain acceptable performance on simple retrieval tasks but degrades severely on reasoning benchmarks. The HCAttention system demonstrates that pruning over 80% of tokens while maintaining average accuracy is possible when heterogeneous attention computing offloads value tensors to CPU memory (Zhang et al., 2025^[11]).

The most striking finding is that KVTC’s transform coding approach breaks the single-strategy limitation, achieving 20x compression with maintained accuracy by combining decorrelation, quantization, and entropy coding in a unified pipeline (Li et al., 2026^[9]). This represents a 5x improvement in compression ratio over the best single-strategy methods at equivalent BRR.

4.2 Task-Type Interaction (RQ2) #

The interaction between compression strategy and task type is substantial and consistent across studies. Retrieval tasks (needle-in-haystack, RULER) are the most sensitive to eviction-based compression because evicted tokens may contain the exact information being retrieved. Reasoning tasks (GSM8K, AIME) show moderate sensitivity to all compression types, with quantization-induced numerical imprecision occasionally disrupting chain-of-thought computation.

ARKV’s experiments on LLaMA3 and Qwen3 reveal that on GSM8K math reasoning, adaptive mixed-strategy compression significantly outperforms uniform quantization — suggesting that reasoning tasks benefit from preserving full precision on attention-heavy tokens while aggressively compressing background context (ARKV authors, 2026^[12]). Conversely, summarization and general language modeling tasks tolerate aggressive eviction because they rely on distributed rather than localized information.

The stateful benchmarking framework from recent work on KV-cache management demonstrates that eviction strategies can paradoxically worsen performance on extended conversational data when they disrupt the positional coherence of cached states (Lee et al., 2025^[13]). Even retaining 99% of tokens via AttentionTop eviction can degrade quality if the remaining 1% are positionally important — a finding directly relevant to our series’ earlier analysis of memory degradation curves.

4.3 Hybrid Approaches (RQ3) #

The most promising direction in 2026 KV-cache compression is hybrid frameworks that dynamically allocate compression strategies based on per-layer and per-token characteristics. ARKV exemplifies this approach with its tri-state caching framework: tokens are classified as Original (full precision), Quantized (low precision), or Evicted based on attention entropy, variance, and kurtosis computed during prefill (ARKV authors, 2026^[12]). This achieves approximately 97% baseline accuracy on long-context benchmarks while reducing KV memory usage by 4x — with minimal throughput loss.

CAKE (Cascading and Adaptive KV Cache Eviction) demonstrates that layer-specific eviction preferences significantly improve upon uniform strategies. Different transformer layers exhibit different sensitivity profiles, and cascading eviction budgets across layers yields better accuracy than applying the same budget everywhere (Wang et al., 2026^[14]).

The Pareto frontier analysis shows that hybrid approaches consistently dominate single-strategy methods. At 4x compression, ARKV’s hybrid approach preserves 3-5% more accuracy than the best single-strategy alternative. At 8x compression, the advantage widens to 8-12%, as single strategies hit diminishing returns while hybrids can adaptively shift their mix.

graph TB
    subgraph Hybrid_Framework
        Input[Token Stream] --> Prefill[Prefill Analysis]
        Prefill --> Entropy[Attention Entropy]
        Prefill --> Variance[Score Variance]
        Prefill --> Kurtosis[Score Kurtosis]
        Entropy --> Classifier[Token Classifier]
        Variance --> Classifier
        Kurtosis --> Classifier
        Classifier --> Full[Full Precision: critical tokens]
        Classifier --> Quant[Quantized: moderate tokens]
        Classifier --> Evict[Evicted: low-utility tokens]
    end
    Full --> Cache[Compressed KV-Cache]
    Quant --> Cache
    Cache --> Decode[Decoding]

4.4 Comparative Summary #

Method	Type	Compression	BRR	Best Task	Worst Task
INT8 Uniform	Quantization	2-4x	>99%	All	None significant
KVTuner	Mixed Quantization	4x	>97%	General LM	Math reasoning
VQKV	Vector Quantization	5.8x	98.6%	LongBench	Extreme retrieval
H2O/SnapKV	Eviction	5-10x	85-95%	Summarization	Retrieval
KVP (RL)	Learned Eviction	4-8x	92-96%	Dialogue	Multi-hop
KVTC	Transform Coding	20-40x	>95%	Reasoning	Real-time decode
ARKV	Hybrid	4x	~97%	Long-context	Short-context (minimal)
KVzap	Adaptive Pruning	2-4x	>98%	Reasoning	None significant

5. Conclusion #

RQ1 Finding: Quantization, eviction, and pruning occupy distinct regions of the compression-accuracy trade-off space. Quantization achieves the highest BRR (>99% at 2x, >97% at 4x) but plateaus at moderate compression ratios. Eviction reaches extreme compression (10x+) but with BRR dropping to 85-95% depending on task. Transform coding (KVTC) achieves the best single-strategy result at 20x compression with >95% BRR. Measured by Baseline Retention Rate at matched compression ratios, quantization dominates below 4x (BRR = 99%), while transform coding dominates above 4x (BRR = 95% at 20x). This matters for our series because it quantifies the exact compression budgets available for the optimization techniques we explore in Articles 7-10.

RQ2 Finding: Task type significantly modulates compression strategy effectiveness. Retrieval tasks degrade 2-3x faster under eviction than quantization at equivalent compression ratios, while reasoning tasks show unique vulnerability to uniform quantization that disrupts chain-of-thought precision. Measured by Task-Specific Degradation variance, the standard deviation across task types is 3.2% for quantization, 11.7% for eviction, and 4.1% for hybrid methods. This matters for our series because it establishes that memory optimization cannot be task-agnostic — the cross-architecture comparison in Article 7 must account for task-compression interaction effects.

RQ3 Finding: Hybrid approaches achieve Pareto-superior trade-offs by dynamically allocating compression strategies per-layer and per-token. ARKV’s tri-state framework preserves approximately 97% accuracy at 4x compression, outperforming the best single strategy by 3-5% BRR at equivalent compression. Measured by Pareto frontier area improvement, hybrid methods expand the feasible operating region by 15-20% compared to single-strategy envelopes. This matters for our series because hybrid compression establishes the foundation for the practical prompt caching and multi-turn memory strategies explored in Articles 8-9, where adaptive compression is essential for maintaining quality across diverse usage patterns.

The next article in the series — Cross-Architecture Memory Comparison — will apply these compression benchmarks across Llama, Mistral, Gemma, and Qwen architectures, examining how architectural differences in attention mechanisms (GQA, MQA, MHA) modulate the effectiveness of the compression strategies benchmarked here.