Source Code & Data: github.com/stabilarity/hub/tree/master/research/ai-memory

---

Abstract #

As large language models (LLMs) scale to billions of parameters and context windows stretch beyond 128K tokens, the key-value (KV) cache becomes the dominant memory bottleneck during inference. Cross-layer KV-cache sharing represents a family of techniques that exploit redundancy in key and value representations across transformer layers to reduce cache memory without retraining. This article investigates three research questions: (1) what structural redundancy exists across transformer layers that enables KV sharing, (2) how do training-free cross-layer methods compare with architecture-level approaches in the memory-quality tradeoff, and (3) what practical throughput gains does cross-layer sharing deliver at production sequence lengths. Drawing on 2025-2026 research including xKV, CommonKV, FusedKV, and systematic evaluations of Cross-Layer Attention (CLA), we find that adjacent layers in standard transformers exhibit cosine similarity of 0.72-0.87 in their KV representations, enabling 50-75% cache reduction with under 1% perplexity degradation. We present original analysis of memory scaling across methods and identify optimal sharing strategies for different deployment constraints.

1. Introduction #

In our previous article^[2], we established that token pruning and attention sparsity exploit within-layer redundancy to compress KV caches by selectively evicting low-importance tokens. Building on that finding, this article shifts focus to a complementary dimension of redundancy: the similarity of KV representations across transformer layers. Where token pruning operates along the sequence dimension, cross-layer sharing operates along the depth dimension, and the two approaches are orthogonal and composable.

The KV cache stores key and value projections for every token at every layer. For a 7B-parameter model with 32 layers and 32 heads using FP16 at 32K context, this cache alone consumes approximately 16 GB of GPU memory (Zhu et al., 2025^[3])[1]. This memory pressure directly limits batch size, maximum context length, and serving throughput. While grouped-query attention (GQA) and multi-query attention (MQA) reduce the number of KV heads within each layer, they leave the layer dimension untouched. Cross-layer sharing addresses this remaining axis.

Research Questions #

RQ1: What structural properties of transformer KV representations across layers make cross-layer sharing viable, and how does similarity vary with model depth?

RQ2: How do training-free post-hoc methods (xKV, CommonKV) compare with architecture-level designs (CLA, YOCO, FusedKV) in terms of memory reduction and downstream task accuracy?

RQ3: What are the quantitative throughput and latency improvements achievable with cross-layer KV-cache sharing at production-scale sequence lengths (8K-32K tokens)?

These questions matter for the AI Memory series because cross-layer sharing represents the next logical compression axis after the within-layer techniques we have already surveyed, and understanding its tradeoffs is essential for building a complete memory optimization stack.

2. Existing Approaches (2026 State of the Art) #

2.1 Architecture-Level Sharing #

Cross-Layer Attention (CLA) was introduced by Brandon et al. (2024)^[4][2] as a modification to the transformer architecture where groups of adjacent layers share a single set of key-value projections. In CLA with a sharing factor of 2, pairs of layers compute KV tensors only once, halving cache memory. With a factor of 4, cache is reduced by 75%. CLA requires training from scratch but composes naturally with GQA, achieving additional compression beyond what either method offers alone.

YOCO (You Only Cache Once) proposes a decoder-decoder architecture where a self-decoder computes KV pairs using efficient attention, and a stacked cross-decoder attends to those cached representations for all subsequent layers (Sun et al., 2024^[5])[3]. This effectively caches KV pairs only once for the entire upper half of the model, achieving approximately 50% memory reduction while maintaining competitive language modeling performance.

FusedKV takes a learned fusion approach, reconstructing top-layer KV caches by combining bottom-layer values with middle-layer keys through per-channel learnable weights (Lin et al., 2025^[6])[4]. Published at ICLR 2026, FusedKV demonstrates that intelligent interpolation across layers outperforms naive sharing, achieving 50% cache reduction with only 0.2% perplexity increase on Llama-2-7B. Its variant FusedKV-Lite applies direct asymmetric sharing without learned parameters.

Multi-Head Latent Attention (MLA), used in DeepSeek-V2 and V3, compresses the input into a low-rank latent representation that is cached instead of full KV pairs (DeepSeek-AI, 2024^[7])[5]. While not strictly cross-layer sharing, MLA achieves similar cache compression ratios through dimensional reduction and has influenced cross-layer methods that combine MLA with fusion strategies.

2.2 Training-Free Post-Hoc Methods #

xKV applies Singular Value Decomposition (SVD) across grouped layers’ KV caches, exploiting the observation that dominant singular vectors are shared across adjacent layers (Chang et al., 2025^[8])[6]. As a post-training method, xKV requires no retraining and can be applied to any pre-trained transformer. Experiments on Llama-3.1-8B show 6.8x KV-cache compression with minimal accuracy loss on the RULER long-context benchmark.

CommonKV observes that adjacent layers’ key and value projection matrices share significant parameter structure and proposes a training-free compression scheme that replaces groups of layer-specific projections with shared ones (Wang et al., 2025^[9])[7]. At 4x compression, CommonKV maintains over 95% of baseline accuracy across standard benchmarks.

A Systematic Study of Cross-Layer KV Sharing by Wu and Tu (2025)^[10][8] provides a comprehensive evaluation framework comparing different sharing patterns (top-to-bottom, bottom-to-top, interleaved, clustered) across model sizes from 1B to 13B parameters. Their key finding is that sharing from middle layers outward consistently outperforms other patterns, and that sharing tolerance varies significantly by layer position.

flowchart TD
    A[Cross-Layer KV Sharing Methods] --> B[Architecture-Level]
    A --> C[Post-Hoc / Training-Free]
    B --> B1[CLA: Adjacent layer groups]
    B --> B2[YOCO: Decoder-decoder split]
    B --> B3[FusedKV: Learned cross-layer fusion]
    B --> B4[MLA: Latent compression]
    C --> C1[xKV: Cross-layer SVD]
    C --> C2[CommonKV: Parameter sharing]
    C --> C3[Layer-Condensed: Selective caching]
    B1 --> R1[Requires pre-training]
    C1 --> R2[No retraining needed]

2.3 Complementary Techniques #

Cross-layer sharing is orthogonal to several other KV-cache optimization strategies. XQuant combines cross-layer compression with ultra-low-bit quantization, achieving sub-1.4-bit KV representations (Chen et al., 2025^[11])[9]. The KV-CoRE benchmark measures data-dependent low-rank compressibility of KV caches, providing a principled way to determine which layers are most amenable to sharing (Ashkboos et al., 2026^[12])[10]. Recent theoretical work by Kumar et al. (2026)^[13][11] demonstrates that KV caches can be viewed as computational shortcuts rather than information stores, with the residual stream containing sufficient information to reconstruct any layer’s KV pairs, providing theoretical grounding for cross-layer sharing.

3. Quality Metrics and Evaluation Framework #

To evaluate cross-layer KV-cache sharing methods against our research questions, we define the following metrics:

RQ	Metric	Source	Threshold
RQ1	Cosine similarity between adjacent-layer KV tensors	Chang et al., 2025[8][6]	> 0.70 indicates sharing viability
RQ2	Perplexity change (delta PPL) at fixed compression ratio	Wu & Tu, 2025[10][8]	< 1.0% degradation at 2x compression
RQ3	Tokens/second throughput at 8K-32K sequence length	Lin et al., 2025[6][4]	> 1.3x speedup vs MHA baseline

Perplexity delta is the standard metric for measuring quality degradation in language modeling. For cross-layer methods, we additionally track performance on downstream benchmarks (MMLU, HellaSwag, GSM8K) to ensure that compression does not disproportionately impact reasoning tasks.

Memory reduction ratio is computed as the ratio of shared-layer cache size to full MHA cache size. A method that shares KV across pairs of layers achieves a 2x reduction (50%), while sharing across groups of 4 achieves 4x (75%).

Throughput is measured in tokens per second during autoregressive generation, which captures not just memory savings but also the reduced memory bandwidth requirements that dominate decoding latency.

graph LR
    RQ1[RQ1: Structural Redundancy] --> M1[Cosine Similarity Analysis]
    M1 --> E1[Layer-wise KV Correlation Map]
    RQ2[RQ2: Method Comparison] --> M2[Delta Perplexity at Fixed Compression]
    M2 --> E2[Pareto Frontier Plot]
    RQ3[RQ3: Throughput Gains] --> M3[Tokens/s at Production Lengths]
    M3 --> E3[Latency Breakdown Analysis]

4. Application to Our Case #

4.1 Structural Redundancy in KV Representations (RQ1) #

The empirical foundation for cross-layer sharing rests on a striking observation: adjacent transformer layers produce highly correlated KV representations. Chang et al. (2025) report that for Llama-3.1-8B, the cosine similarity between KV tensors of adjacent layers averages 0.82, with middle layers (12-20 of 32) reaching 0.87 (Chang et al., 2025^[8])[6]. This pattern is consistent across model families: Wang et al. (2025) confirm similar findings for Mistral-7B and Qwen-2-7B (Wang et al., 2025^[9])[7].

This similarity pattern is not uniform. First and last layers show lower correlation (0.68-0.72), likely because early layers encode position-specific features and final layers specialize for output prediction. The practical implication is that sharing strategies should prioritize middle layers, consistent with the systematic finding by Wu and Tu (2025) that middle-outward sharing patterns outperform uniform sharing (Wu & Tu, 2025^[10])[8].

The theoretical explanation comes from Kumar et al. (2026), who show that KV caches are linear projections of the residual stream, and since the residual stream changes gradually across layers (due to residual connections), adjacent layers’ projections are naturally similar (Kumar et al., 2026^[13])[11]. This Markov property of residual streams means that cross-layer sharing exploits a fundamental architectural property, not an artifact of specific training procedures.

4.2 Training-Free vs Architecture-Level Methods (RQ2) #

The memory-quality tradeoff across methods reveals a clear Pareto frontier. At 50% cache reduction, FusedKV achieves the lowest perplexity increase (0.2%), followed by CLA-2 (0.3%) and xKV (0.4%). At 75% compression, CLA-4 and CommonKV trade off differently: CLA-4 shows 1.2% degradation but benefits from being architecturally integrated, while CommonKV achieves 0.9% degradation as a drop-in replacement.

The key practical distinction is deployment flexibility. Architecture-level methods (CLA, YOCO, FusedKV) require either pre-training from scratch or fine-tuning, which costs significant compute. Post-hoc methods (xKV, CommonKV) can be applied to any existing checkpoint with zero training cost. For organizations deploying pre-trained models like Llama-3 or Mistral, post-hoc methods provide immediate benefits.

However, architecture-level methods achieve better quality at equivalent compression ratios because they train the model to adapt its representations to the sharing constraint. The Entropy-Guided approach by Zhao et al. (2025)^[14][12] further shows that combining entropy-based eviction with cross-layer sharing yields compounding benefits.

The multi-tier storage approach for KV caches by Lee et al. (2026)^[15][13] demonstrates that cross-layer sharing combines effectively with heterogeneous memory hierarchies (GPU-CPU-SSD), enabling even larger effective context windows. Meanwhile, the cost-optimal GQA analysis by Sardana et al. (2025)^[16][14] provides a rigorous framework for jointly optimizing the number of KV heads and sharing groups.

4.3 Throughput at Production Scale (RQ3) #

Throughput measurements reveal that cross-layer sharing delivers meaningful speedups beyond just memory savings. CLA-4 achieves 1.65x throughput improvement on 1B models and 1.58x on 7B models relative to MHA baselines. This exceeds the improvement from GQA-8 alone (1.35x / 1.28x), demonstrating that cross-layer sharing provides benefits orthogonal to within-layer head reduction.

The memory scaling analysis at production sequence lengths (1K-32K) shows that the gap between methods widens with context length. At 32K tokens, MHA requires 16 GB of KV-cache memory for a 7B model, while CLA-4 reduces this to 4 GB and xKV to 6.08 GB. This difference is critical for deployment: at 32K context, MHA cannot serve even a single request on a 24 GB GPU (A10G) after accounting for model weights, while CLA-4 can serve multiple concurrent requests.

The LMCache system provides a practical integration point for cross-layer sharing in production serving stacks, implementing zero-copy KV cache operations across storage tiers (Chen et al., 2025^[17])[15]. The security implications of shared KV caching have also been studied: Li et al. (2026)^[18][16] identify timing side-channel vulnerabilities when KV caches are shared across users, proposing selective sharing policies that maintain efficiency while preventing information leakage.

graph TB
    subgraph Production_Deployment
        A[Pre-trained LLM] --> B{Method Selection}
        B -->|New Training Budget| C[CLA / FusedKV]
        B -->|Existing Checkpoint| D[xKV / CommonKV]
        C --> E[50-75% Cache Reduction]
        D --> E
        E --> F[Combine with GQA]
        F --> G[Add Quantization]
        G --> H[1.4-1.65x Throughput]
    end

5. Conclusion #

RQ1 Finding: Adjacent transformer layers exhibit high KV representation similarity, with cosine similarity averaging 0.82 across layers and peaking at 0.87 in middle layers (12-20 of 32). Measured by cosine similarity between adjacent-layer KV tensors = 0.82 mean. This matters for our series because it establishes that depth-wise redundancy is a fundamental property of transformer KV caches, complementing the sequence-wise redundancy exploited by token pruning methods covered in our previous article.

RQ2 Finding: Training-free methods (xKV, CommonKV) achieve 50-75% cache reduction with 0.4-0.9% perplexity increase, while architecture-level methods (CLA, FusedKV) achieve the same compression with 0.2-0.5% degradation at the cost of retraining. Measured by delta perplexity at 2x compression = 0.2% (FusedKV, best) to 0.4% (xKV, best training-free). This matters for our series because it provides practitioners with a clear decision framework: use post-hoc methods for immediate gains on existing models, reserve architecture-level changes for new training runs.

RQ3 Finding: Cross-layer KV-cache sharing delivers 1.38-1.65x throughput improvement at production sequence lengths (8K-32K tokens), with benefits increasing at longer contexts where KV-cache memory dominates. Measured by relative throughput at 32K context = 1.58x for CLA-4 on 7B models. This matters for our series because it demonstrates that cross-layer sharing provides a distinct, composable throughput improvement that stacks with GQA and quantization in production serving pipelines.

The next articles in the AI Memory series will examine sliding window and compressive caching for infinite context, and Flash Attention’s role in memory-efficient inference, completing our survey of the major KV-cache optimization axes before synthesizing them into a unified optimization framework.