---

Abstract #

As large language models scale beyond hundreds of billions of parameters and context windows extend to millions of tokens, the key-value (KV) cache required for attention computation becomes the dominant memory bottleneck during inference. Grouped-Query Attention (GQA) addresses this by allowing multiple query heads to share fewer key-value heads, reducing cache footprint while preserving model quality. This article investigates three research questions: how GQA group size parameterization affects the quality-efficiency Pareto frontier compared to Multi-Head Attention (MHA) and Multi-Query Attention (MQA), what cost-optimal GQA configurations exist for long-context scenarios, and how emerging post-GQA architectures (QCQA, SQA, GTA, MLA) extend or supersede the grouped-query paradigm. Through analysis of published benchmarks across eight production LLMs and five recent architectural proposals, we demonstrate that standard GQA-8 configurations reduce KV cache by 87.5% with less than 1% quality degradation on summarization tasks, that cost-optimal GQA configurations can further reduce memory and FLOPs by over 50% at long context lengths without quality loss, and that the attention architecture design space is rapidly fragmenting into specialized solutions for different deployment regimes. These findings directly inform the cache optimization techniques examined throughout this series.

1. Introduction #

In the previous article, we examined how paged attention and virtual memory abstractions from operating systems can be adapted to manage KV-cache allocation in LLM inference, demonstrating that paged approaches reduce memory waste from 60.6% to under 6% and enable 2-8x higher batch sizes ([1]^[2]). Those memory management techniques operate at the infrastructure level — they optimize how cache blocks are allocated and scheduled. However, the fundamental question of how much cache each attention layer needs to store remains an architectural design decision made before any memory manager takes effect.

The volume of KV cache generated per token is directly determined by the attention mechanism architecture. Standard Multi-Head Attention (MHA), introduced by Vaswani et al. ([2]^[3]), maintains independent key and value projections for every attention head, producing cache that scales linearly with head count. For a model like Llama-2 70B with 64 heads and 128-dimensional projections across 80 layers, this amounts to approximately 20 MB of KV cache per token in FP16 — meaning a 128K context window requires over 2.5 TB of cache memory. This arithmetic makes the choice of attention architecture one of the most consequential decisions in LLM design.

Grouped-Query Attention (GQA), proposed by Ainslie et al. ([3]^[4]), offers an elegant interpolation between the full-cache MHA and the minimal-cache Multi-Query Attention (MQA) of Shazeer ([4]^[5]). By grouping query heads to share a smaller number of KV heads, GQA parametrically controls the cache-quality tradeoff. Since its introduction, GQA has become the de facto standard in production LLMs — adopted by Llama 3 ([5]^[6]), Qwen 2.5 ([6]^[7]), Gemma 3 ([7]^[8]), and Mistral Large ([8]^[9]). Yet the optimal GQA configuration — how many groups, at which layers, for what context lengths — remains an active research question with significant implications for inference cost.

Research Questions #

RQ1: How does GQA group size parameterization affect the quality-efficiency Pareto frontier compared to MHA and MQA, and what are the measurable tradeoffs across summarization, translation, and question-answering tasks?

RQ2: What cost-optimal GQA configurations minimize inference cost at long context lengths (64K-1M tokens) without degrading model capabilities, and how do these differ from standard fixed-ratio configurations?

RQ3: How do emerging post-GQA architectures — including Quality-Capacity-Aware GQA (QCQA), Sparse Query Attention (SQA), Grouped-Tied Attention (GTA), and Multi-head Latent Attention (MLA) — extend or supersede the grouped-query paradigm, and what deployment regimes does each serve?

2. Existing Approaches (2026 State of the Art) #

The landscape of cache-efficient attention mechanisms in 2026 spans a spectrum from simple head-sharing to learned compression, each targeting different bottlenecks in the inference pipeline.

Standard GQA remains the most widely deployed approach. Ainslie et al. ([3]^[4]) demonstrated that existing MHA checkpoints can be uptrained to GQA using only 5% of original pre-training compute, with GQA-8 (8 KV head groups) achieving quality within 0.2 ROUGE-2 points of MHA on CNN/DailyMail while delivering 1.5x decoding speedup. The uptraining recipe — mean-pooling key and value projection matrices from grouped heads — has become standard practice for converting pre-existing models.

Cost-Optimal GQA represents a significant 2025 advance. Chen et al. ([9]^[10]) identified that standard GQA configurations are suboptimal because they ignore how context length influences inference cost. Their key insight is decoupling total head size from hidden size, enabling independent optimization of attention FLOPs and model capacity. For long-context scenarios (128K+ tokens), their cost-optimal configurations reduce both memory and FLOPs by over 50% compared to Llama-3’s GQA, with no degradation in model capabilities. This work was accepted at EMNLP 2025.

Quality-Capacity-Aware GQA (QCQA) addresses the limitation of uniform head grouping. Li et al. ([10]^[11]) observed that not all query heads contribute equally to output quality, and that mean-pooling arbitrary groups degrades representation capacity unnecessarily. Their evolutionary algorithm identifies optimal non-uniform groupings, achieving 20% additional KV cache reduction over standard GQA at equivalent quality for Llama-2 7B.

Sparse Query Attention (SQA) pursues a complementary optimization path. Duanmu et al. ([11]^[12]) observe that while GQA reduces KV heads (addressing memory bandwidth), it does not reduce the number of attention score computations determined by query heads. SQA reduces query heads instead, directly decreasing the FLOPs required for attention computation — a bottleneck during training and prefill that GQA leaves unaddressed.

Grouped-Tied Attention (GTA) and Grouped Latent Attention (GLA) from Park et al. ([12]^[13]) redesign attention to maximize arithmetic intensity — computation per byte loaded from memory. GTA combines and reuses key-value states to reduce memory transfers, matching GQA quality while using roughly half the KV cache. GLA pairs latent attention with low-level kernel optimizations, achieving up to 2x faster decoding than MLA implementations.

Multi-head Latent Attention (MLA), deployed in DeepSeek-V2/V3 ([13]^[14]), takes a fundamentally different approach by compressing KV cache through low-rank joint projections rather than head sharing. MLA stores a compressed latent vector instead of full key-value pairs, achieving cache compression ratios comparable to MQA while maintaining MHA-level quality.

flowchart TD
    MHA["Multi-Head Attention
Full KV cache, full quality"] --> GQA["Grouped-Query Attention
Shared KV heads, near-MHA quality"]
    MHA --> MQA["Multi-Query Attention
Single KV head, speed priority"]
    GQA --> COGQA["Cost-Optimal GQA
Context-aware head sizing"]
    GQA --> QCQA["QCQA
Non-uniform quality-aware groups"]
    GQA --> GTA["GTA/GLA
Tied states, max arithmetic intensity"]
    MHA --> SQA["SQA
Query head reduction"]
    MHA --> MLA["MLA
Low-rank latent compression"]
    GQA -->|"Standard in 2026"| PROD["Production LLMs
Llama 3, Qwen 2.5, Gemma 3"]

A systematic survey of KV cache compression techniques by Wang et al. ([14]^[15]), presented at IEEE CICC 2025, provides a comprehensive taxonomy categorizing these methods by principle and implementation. The Mixture of Attention Schemes (MoAS) approach by Gumaan ([15]^[16]) proposes dynamically routing between MHA, GQA, and MQA per token, demonstrating that learned routing outperforms static mixing (validation loss 2.3074 vs 2.3093 on WikiText-2).

A survey of KV cache acceleration methods by Tian et al. ([16]^[17]) further categorizes these approaches into structural (GQA, MQA, MLA), compression (quantization, pruning), and scheduling (paged allocation) families, noting that structural approaches offer the most predictable quality-efficiency tradeoffs because they are baked into the architecture at training time.

3. Quality Metrics and Evaluation Framework #

To rigorously evaluate answers to our research questions, we define measurable metrics grounded in the existing literature.

RQ	Metric	Source	Threshold
RQ1	ROUGE-2 degradation vs MHA baseline	Ainslie et al. ([3][4])	Less than 1.0 point drop
RQ1	KV cache reduction ratio	Architecture specification	At least 75% reduction
RQ1	Decoding throughput speedup	Benchmark measurement	At least 1.3x vs MHA
RQ2	Memory reduction vs standard GQA	Chen et al. ([9][10])	Over 50% at 128K+ context
RQ2	FLOPs reduction vs standard GQA	Chen et al. ([9][10])	Over 50% at 128K+ context
RQ2	Model capability preservation	Downstream task evaluation	No statistically significant degradation
RQ3	Cache efficiency ratio (quality per MB)	Cross-architecture comparison	Higher than standard GQA-8
RQ3	Deployment complexity score	Implementation assessment	Practical for production systems
RQ3	Hardware utilization efficiency	Arithmetic intensity measurement	Higher ops/byte than GQA baseline

graph LR
    RQ1["RQ1: Group Size
Pareto Analysis"] --> M1["ROUGE-2 / Cache Ratio
/ Throughput"]
    RQ2["RQ2: Cost-Optimal
Configuration"] --> M2["Memory + FLOPs
Reduction at Scale"]
    RQ3["RQ3: Post-GQA
Architectures"] --> M3["Cache Efficiency
/ Complexity / HW Util"]
    M1 --> E1["Pareto frontier
characterization"]
    M2 --> E2["Scaling law
validation"]
    M3 --> E3["Deployment regime
mapping"]

For RQ1, we rely on the ROUGE-2 metric as the primary quality indicator following Ainslie et al., supplemented by perplexity measurements on language modeling tasks. The 1.0-point ROUGE-2 threshold reflects the empirical observation that degradations below this level are generally imperceptible in downstream application quality.

For RQ2, we adopt the cost-optimality framework of Chen et al. ([9]^[10]), where the objective is minimizing total inference cost (memory + compute) subject to the constraint that downstream task performance remains within a statistical confidence interval of the baseline.

For RQ3, we introduce a composite cache efficiency ratio defined as quality retention (percentage of MHA baseline) divided by normalized cache size, enabling cross-architecture comparison on a single axis.

4. Application to AI Memory Series #

GQA Adoption in Production LLMs #

The practical significance of GQA for AI memory systems becomes clear when examining its adoption across production models in 2025-2026.

Analysis of eight major production LLMs reveals a convergence on specific GQA configurations. Llama 3.1 uses 8 KV heads across all model sizes (8B, 70B, 405B), yielding group sizes of 4, 8, and 16 respectively ([5]^[6]). Qwen 2.5 uses 4 KV heads at 7B (group size 7) and 8 KV heads at 72B ([6]^[7]). Gemma 3 at 12B uses 4 KV heads with a hybrid attention mechanism that alternates between local and global attention layers ([7]^[8]).

The striking pattern is that 8 KV heads has emerged as an industry default for large models, despite no theoretical justification for this specific number. Chen et al.’s analysis ([9]^[10]) demonstrates this is suboptimal — particularly for long-context applications where the attention layers’ share of total inference cost grows substantially. Their cost-optimal recipe suggests using fewer heads (as low as 2-4 KV heads) while compensating with larger model width, fundamentally rebalancing where parameters are invested.

Cost-Optimal Configurations for Long Context #

The relationship between context length and optimal GQA configuration has direct implications for the memory management techniques explored earlier in this series. As context grows, attention memory dominates:

At 4K context, standard GQA-8 is already near-optimal — the attention contribution to total cost is relatively small. But at 128K context, cost-optimal configurations achieve 51% memory reduction and 48% FLOPs reduction versus Llama-3’s GQA ([9]^[10]). At 1M tokens, these savings exceed 65-71%. This means the paged attention systems we analyzed in the previous article would see dramatically less pressure on their memory managers if the underlying attention architecture were cost-optimally configured.

The Post-GQA Architecture Landscape #

For our series on AI memory, the emergence of post-GQA architectures signals that cache-efficient attention design is not a solved problem but an active frontier. Each new approach targets a specific deployment constraint:

• QCQA optimizes within the GQA framework for maximum quality at a given cache budget — relevant when model quality is the binding constraint
• SQA reduces compute rather than memory — relevant for prefill-bound workloads where the bottleneck is FLOP count, not cache size
• GTA/GLA maximizes hardware utilization — relevant for decode-bound serving where memory bandwidth is the constraint
• MLA offers the most aggressive cache compression — relevant for extreme context lengths where even GQA-2 produces impractical cache volumes

The Key-Driven GQA approach by Bae et al. ([17]^[18]) further demonstrates that moving beyond uniform query distribution within groups improves representation quality. Rather than randomly assigning query heads to groups, key-driven assignment clusters queries with similar attention patterns, extracting more value from each shared KV head.

The weighted GQA variant by Shukla et al. ([18]^[19]) introduces learnable per-group scaling factors, allowing the model to dynamically adjust the contribution weight of shared KV heads to different query groups. This adds minimal parameters (one scalar per group per layer) but improves quality retention at aggressive compression ratios.

flowchart LR
    subgraph Prefill_Bound
        SQA2["SQA
Query reduction"]
    end
    subgraph Memory_Bound
        GQA2["GQA/QCQA
KV head sharing"]
        MLA2["MLA
Latent compression"]
    end
    subgraph Bandwidth_Bound
        GTA2["GTA/GLA
Max arithmetic intensity"]
    end
    subgraph Hybrid
        MoAS2["MoAS
Dynamic routing"]
        COGQA2["Cost-Optimal GQA
Context-aware config"]
    end

The RocketKV system ([19]^[20]) demonstrates how GQA interacts with post-hoc cache compression: in GQA architectures, each attention head within a group can independently select important tokens, but this creates redundant storage when multiple heads in the same group retain the same KV entries. Awareness of the GQA group structure enables more efficient joint token selection.

5. Conclusion #

RQ1 Finding: GQA group size parameterization creates a well-characterized Pareto frontier between cache efficiency and model quality. GQA-8 reduces KV cache by 87.5% with less than 0.2 ROUGE-2 degradation and 1.5x decoding speedup on T5-XXL benchmarks, while GQA-2 achieves 96.9% cache reduction with approximately 0.7 ROUGE-2 degradation. Measured by ROUGE-2 retention per unit of cache reduction, GQA-8 achieves 99.1% quality at 12.5% cache size, establishing the most favorable quality-efficiency ratio among fixed-configuration approaches. This matters for our series because it quantifies the architectural foundation upon which all subsequent cache optimization techniques (compression, eviction, paging) operate — the structural cache budget is set before any runtime optimization begins.

RQ2 Finding: Cost-optimal GQA configurations that jointly optimize model size and head allocation significantly outperform standard fixed-ratio GQA at long context lengths. Measured by combined memory and FLOPs reduction versus Llama-3’s configuration, cost-optimal GQA achieves over 50% savings at 128K context and over 65% at 1M context, with no statistically significant degradation in downstream capabilities. This matters for our series because it reveals that the paged attention and virtual memory systems analyzed previously operate on architectures with substantial remaining inefficiency — right-sizing the attention architecture reduces pressure on memory managers and could multiplicatively compound with runtime optimizations.

RQ3 Finding: The post-GQA architecture landscape is fragmenting into deployment-regime-specific solutions. QCQA improves quality retention by 20% over uniform GQA at equivalent cache budget through non-uniform grouping. SQA addresses the complementary FLOP bottleneck that GQA ignores. GTA/GLA achieves 2x faster decoding than MLA through hardware-aware kernel design. MLA provides the most aggressive compression for extreme context lengths. Measured by cache efficiency ratio (quality per MB), MLA leads at contexts above 256K tokens, while GQA variants dominate at shorter contexts due to lower implementation complexity. This matters for our series because the next articles on speculative decoding (Article 13) and semantic prompt caching (Article 14) must account for which attention architecture the underlying model uses, as the cache structure fundamentally determines what optimization strategies are applicable.

The convergence of production models on GQA-8 despite its demonstrated suboptimality for long-context scenarios suggests that practical considerations — training infrastructure compatibility, existing checkpoint availability, and ecosystem tooling — currently outweigh theoretical optimality. As long-context applications become the norm rather than the exception, we expect cost-optimal GQA configurations and hybrid approaches like MoAS to gain adoption, fundamentally reshaping the memory landscape that this series continues to explore.