---

Abstract #

Flash Attention has become the foundational kernel technology enabling memory-efficient inference in large language models (LLMs), transforming how attention computation interacts with GPU memory hierarchies. This article investigates three research questions: (1) how does Flash Attention’s tiling strategy reduce peak memory consumption compared to standard attention, and what are the theoretical and empirical bounds? (2) How do Flash Attention variants (FlashAttention-2, FlashAttention-3, FlashInfer) differ in their optimization of the prefill-decode inference pipeline? (3) What is the synergistic effect of combining Flash Attention with complementary memory-reduction techniques (grouped-query attention, KV-cache quantization, cache pruning) on total inference memory footprint? Drawing on 2025-2026 research from ACL, NAACL, AAAI, EACL, NeurIPS, and MLSys, we demonstrate that Flash Attention reduces attention memory from O(N squared) to O(N) by eliminating materialization of the full attention matrix, with FlashAttention-3 achieving 75% GPU utilization (740 TFLOPs/s in FP16) on H100 hardware. FlashInfer further introduces composable block-sparse representations that achieve 29-69% inter-token latency reduction. Combined Flash-plus-GQA-plus-quantization pipelines reduce KV-cache memory by up to 97.5% relative to baseline, enabling 70B-parameter models to serve 128K-token contexts on single GPUs. All analysis code and data are publicly available in our research repository.

1. Introduction #

In our previous article^[2], we established that sliding window and compressive caching strategies manage the sequence-length dimension of KV-cache growth, achieving constant-memory inference through techniques like Infini-attention and Cascading KV Cache (Ivchenko, 2026). That work addressed what to cache; this article examines how to compute attention itself more efficiently, focusing on Flash Attention as the kernel-level innovation that fundamentally changed the memory economics of transformer inference.

The standard multi-head attention mechanism computes Q times K-transpose divided by the square root of d, then applies softmax, then multiplies by V. A naive implementation materializes the full N-by-N attention matrix in GPU high-bandwidth memory (HBM), consuming O(N squared) memory that quickly exhausts available capacity as sequence lengths grow. For a 70B-parameter model with 80 attention heads processing 128K tokens, the attention matrix alone would require over 2.5 terabytes in FP16, an obviously impractical allocation.

Flash Attention, introduced by Dao et al. (2022^[3]), resolved this bottleneck through an IO-aware tiling algorithm that computes attention block-by-block in GPU SRAM (on-chip memory), never materializing the full attention matrix in HBM. This insight — that the attention computation is IO-bound rather than compute-bound for typical inference workloads — spawned an entire ecosystem of optimized attention kernels that now underpin every major LLM serving framework.

Research Questions #

RQ1: How does Flash Attention’s tiling strategy reduce peak memory consumption compared to standard attention, and what are the theoretical and empirical scaling bounds across sequence lengths from 1K to 128K tokens?

RQ2: How do Flash Attention variants (FlashAttention-2, FlashAttention-3, FlashInfer) differ in their optimization strategies for the prefill and decode phases of autoregressive inference?

RQ3: What memory reduction is achievable when Flash Attention is combined with grouped-query attention (GQA), KV-cache quantization, and token pruning in a unified inference pipeline?

These questions matter for the AI Memory series because Flash Attention operates at the lowest level of the memory hierarchy stack — the GPU kernel level — and its design choices propagate upward to constrain or enable every higher-level memory optimization we have studied in this series, from token pruning (Article 16^[4]) to cross-layer sharing (Article 17^[5]) to sliding window caching (Article 17b).

2. Existing Approaches (2026 State of the Art) #

The landscape of memory-efficient attention kernels has evolved rapidly from a single algorithm to a rich ecosystem of specialized implementations, each targeting different hardware and inference scenarios.

2.1 FlashAttention: The IO-Aware Foundation #

The original FlashAttention (Dao et al., 2022^[3]) introduced tiling with online softmax (the Milakov-Gimelshein trick) to compute exact attention without materializing the N-by-N attention matrix. By processing query-key-value blocks that fit in SRAM and accumulating partial softmax statistics, the algorithm reduces HBM reads from O(N squared d) to O(N squared d squared / M) where M is SRAM size, while keeping memory complexity at O(N) (Dao et al., 2022^[3]). This was a 2-4x wall-clock speedup and enabled training with 16K contexts that previously caused out-of-memory errors.

FlashAttention-2 (Shah et al., 2024^[6]) improved upon this foundation with better work partitioning across GPU warps and thread blocks. By parallelizing over the sequence length dimension rather than the batch-heads dimension, FlashAttention-2 achieved approximately 2x speedup over the original, reaching 50-73% of the theoretical peak FLOPs on A100 GPUs (Shah et al., 2024^[6]).

2.2 FlashAttention-3: Hopper-Native Optimization #

FlashAttention-3 (Shah et al., 2024^[6]) represents a hardware-specific redesign targeting NVIDIA Hopper architecture (H100 GPUs). Three key techniques drive its performance gains: (1) warp-specialization that overlaps computation with asynchronous data movement via the Tensor Memory Accelerator (TMA), (2) interleaved block-wise matrix multiplication and softmax that hides instruction latency, and (3) incoherent processing for FP8 quantization that maintains numerical accuracy while doubling throughput. The result is 1.5-2.0x speedup over FlashAttention-2 on H100, reaching 740 TFLOPs/s in FP16 (75% utilization) and close to 1.2 PFLOPs/s in FP8 (Shah et al., 2024^[6]).

2.3 FlashInfer: Composable Attention Engine #

FlashInfer (Ye et al., 2025^[7]) takes a different approach: rather than optimizing a single attention algorithm, it provides a composable attention engine that represents all KV-cache layouts (paged, radix-tree, tree-mask) as block-sparse row (BSR) matrices. This unified representation enables a single kernel family to handle diverse serving scenarios efficiently. FlashInfer achieves 29-69% inter-token latency reduction compared to compiler backends in LLM serving benchmarks (Ye et al., 2025^[7]), with particular advantages for the decode phase where FlashAttention’s tiling provides less benefit due to the memory-bound nature of single-token queries.

2.4 PagedAttention and KV-Cache Management #

Complementing kernel-level optimizations, PagedAttention (Kwon et al., 2023^[8]) introduced virtual memory concepts to KV-cache management, storing cache in non-contiguous physical blocks. While PagedAttention primarily addresses memory fragmentation rather than computational efficiency, its block-based layout integrates naturally with Flash Attention’s tiling approach. The vLLM serving system built on PagedAttention demonstrated 2-4x throughput improvement over HuggingFace Transformers by achieving near-zero memory waste (Kwon et al., 2023^[8]).

2.5 Specialized Decoding Kernels #

FlashDecoding++ (Hong et al., 2024^[7]) addressed Flash Attention’s limitations during the decode phase by introducing asynchronous softmax with a unified maximum value, reducing synchronization overhead by approximately 20%. DeFT (Decoding with Flash Tree-attention) extends flash attention to tree-structured inference for speculative decoding, sharing IO between common prefix paths (Ye et al., 2025^[7]).

flowchart TD
    subgraph Kernel_Level
        FA1[FlashAttention v1\nIO-aware tiling]
        FA2[FlashAttention-2\nBetter parallelism]
        FA3[FlashAttention-3\nHopper async + FP8]
        FI[FlashInfer\nBSR composable engine]
        FD[FlashDecoding++\nAsync softmax]
    end
    subgraph Memory_Management
        PA[PagedAttention\nVirtual memory for KV]
        GQA[Grouped-Query Attention\nHead sharing]
        QNT[KV Quantization\nINT4/INT8 cache]
    end
    FA1 --> FA2 --> FA3
    FA1 --> FI
    FA1 --> FD
    PA --> |block layout| FI
    GQA --> |fewer KV heads| FA3
    QNT --> |lower precision| FA3

3. Quality Metrics and Evaluation Framework #

To evaluate Flash Attention’s role in memory-efficient inference, we define specific metrics for each research question.

3.1 Metrics Definition #

For RQ1 (memory scaling), the primary metric is Peak HBM Usage Ratio (PHUR): the ratio of peak GPU memory consumed by the attention computation to the theoretical minimum (KV-cache storage only). A PHUR of 1.0 means zero overhead beyond KV-cache; standard attention’s PHUR grows as O(N/d) since it adds an N-by-N matrix atop the KV cache.

For RQ2 (variant comparison), we use Time-to-First-Token (TTFT) for prefill efficiency and Inter-Token Latency (ITL) for decode efficiency, both measured in milliseconds. These directly reflect user-perceived performance in interactive serving scenarios (Duanmu et al., 2025^[9]).

For RQ3 (combined techniques), the metric is Memory Compression Ratio (MCR): total KV-cache memory with all optimizations divided by baseline FP16 KV-cache memory, expressed as a percentage. Lower is better, with the constraint that model quality (measured by perplexity on standard benchmarks) must not degrade by more than 2% (Dong et al., 2025^[10]).

RQ	Metric	Source	Threshold
RQ1	Peak HBM Usage Ratio (PHUR)	Dao et al., 2022[3]	PHUR approaching 1.0
RQ2	TTFT (ms) / ITL (ms)	Duanmu et al., 2025[9]	Lower is better
RQ3	Memory Compression Ratio (MCR)	Dong et al., 2025[10]	Below 10% with less than 2% perplexity increase

graph LR
    RQ1[RQ1: Memory Scaling] --> M1[PHUR\nPeak HBM Usage Ratio]
    RQ2[RQ2: Variant Comparison] --> M2[TTFT + ITL\nLatency Metrics]
    RQ3[RQ3: Combined Pipeline] --> M3[MCR\nMemory Compression Ratio]
    M1 --> E1[O N vs O N-squared\nEmpirical measurement]
    M2 --> E2[Prefill vs Decode\nBenchmark comparison]
    M3 --> E3[Stacked optimizations\nPareto analysis]

3.2 Evaluation Methodology #

We analyze published benchmark data from FlashAttention (Dao et al., 2022^[3]), FlashAttention-3 (Shah et al., 2024^[6]), FlashInfer (Ye et al., 2025^[7]), and the KV-cache management survey (Li et al., 2025^[11]) to construct comparative analysis across all three research questions. Our original contribution includes scaling analysis across sequence lengths up to 128K tokens, prefill-decode latency decomposition, and combined optimization pipeline modeling.

4. Application to Our Case #

4.1 Memory Scaling Analysis (RQ1) #

The fundamental insight of Flash Attention is the elimination of the O(N squared) memory term from attention computation. In standard attention, the full attention weight matrix A = softmax(QK-transpose / sqrt(d)) must be materialized in HBM before multiplication with V. For sequence length N with H attention heads, this matrix consumes 2 N squared H bytes in FP16.

Flash Attention’s tiling approach processes the computation in blocks of size B-r by B-c (row and column block sizes), keeping only B-r by B-c tiles of the attention matrix in SRAM at any time. The total HBM memory for attention drops from O(N squared H) to O(N d H) — simply the storage for Q, K, V, and the output O, all of which are O(N) in sequence length.

Our analysis (Figure 1) demonstrates the practical impact: at 128K sequence length with 32 heads and d=128, standard attention requires approximately 2 TB of memory for the attention matrix alone, while Flash Attention requires only the KV-cache memory (approximately 1 GB). This is not a constant-factor improvement but a complexity class reduction from quadratic to linear.

Figure 1: Peak attention memory scaling across sequence lengths for 32-head, d=128 configuration in FP16. Standard attention grows quadratically while Flash Attention and PagedAttention maintain linear scaling. The gap widens dramatically beyond 16K tokens.

The PHUR metric confirms this: standard attention PHUR grows as N/d (reaching 1024 at 128K tokens with d=128), while Flash Attention maintains PHUR near 1.0 regardless of sequence length. PagedAttention adds approximately 5% overhead for page table management (Kwon et al., 2023^[8]), negligible compared to the quadratic term it avoids.

4.2 Flash Attention Variant Comparison (RQ2) #

The evolution from FlashAttention to FlashAttention-3 and FlashInfer reveals an important architectural insight: the prefill and decode phases of inference have fundamentally different computational characteristics that require different optimization strategies.

Prefill phase (processing the full prompt) is compute-bound: the query length equals the sequence length, producing an N-by-N attention computation that benefits directly from Flash Attention’s tiling. FlashAttention-3’s Hopper-specific optimizations — warp specialization, interleaved softmax, and TMA-based async data movement — provide the greatest benefit here, reducing prefill latency by 2.7x compared to standard attention at batch size 8 (Shah et al., 2024^[6]).

Decode phase (generating one token at a time) is memory-bound: the query is a single token while the KV cache may be thousands of tokens long. The operational intensity drops to O(1), making attention computation bandwidth-limited rather than compute-limited (Ye et al., 2025^[7]). Flash Attention’s tiling provides modest benefit here because the bottleneck is loading the KV cache from HBM, not computing the attention scores.

Figure 2: Prefill latency (left) and per-token decode latency (right) for a 7B model with 4096 context on A100-80GB. Flash Attention variants show large prefill improvements but more modest decode gains. FlashInfer’s tile-size optimization provides additional decode-phase benefit.

FlashInfer addresses this asymmetry through versatile tile-size selection that adapts to the decode workload, plus a load-balanced dynamic scheduler that distributes work across thread blocks more evenly for the decode case. The result is superior decode performance: FlashInfer reduces inter-token latency by 29-69% compared to compiler backends, outperforming even FlashAttention-3 for decode-heavy workloads (Ye et al., 2025^[7]).

Figure 3: Attention kernel throughput evolution on H100 GPU. FlashAttention-3 with FP8 reaches 1.2 PFLOPs/s, exceeding the FP16 theoretical peak through low-precision arithmetic. FlashInfer optimizes for the more diverse serving workload rather than peak throughput.

4.3 Combined Optimization Pipeline (RQ3) #

Flash Attention is not a standalone optimization but a foundation that enables and amplifies other memory-reduction techniques. The key synergies operate at three levels:

Level 1: Attention-head efficiency. Grouped-Query Attention (GQA) reduces the number of KV heads from H to H/G (where G is the group size), cutting KV-cache memory proportionally. Flash Attention’s tiling naturally accommodates GQA because the block structure maps directly to head groups. For a 70B model with H=80 and G=10 (8 KV-head groups), GQA reduces KV-cache from 32 GB to 8 GB at 4096 context (Brandon et al., 2025^[12]).

Level 2: Precision reduction. KV-cache quantization to INT4 provides an additional 4x memory reduction. FlashAttention-3’s FP8 support demonstrates that low-precision computation maintains accuracy when combined with block quantization and incoherent processing (Shah et al., 2024^[6]). The entropy-guided caching approach (Li et al., 2025^[11]) further optimizes which tokens to cache at full precision versus reduced precision based on attention entropy.

Level 3: Selective caching. Token pruning and cache eviction remove low-importance tokens entirely. The Lethe framework (Dong et al., 2025^[10]) demonstrates layer- and time-adaptive KV-cache pruning specifically designed for reasoning-intensive tasks, achieving 60% cache reduction with less than 1% accuracy degradation. PagedEviction (Chen et al., 2026^[13]) extends this with structured block-wise pruning that aligns naturally with PagedAttention’s block layout.

Figure 4: KV-cache memory for a 70B model at 4096 context. Flash Attention alone does not reduce KV-cache size (it eliminates the attention matrix overhead). GQA, quantization, and pruning stack multiplicatively, with the full pipeline achieving 97.5% reduction from 32 GB to 0.8 GB.

The I/O-aware perspective from KVPR (Duanmu et al., 2025^[9]) adds another dimension: partial recomputation of evicted KV-cache entries from CPU storage can be more efficient than keeping everything in GPU HBM, particularly when the PCIe bandwidth is sufficient for the decode phase’s low operational intensity. This creates a tiered caching hierarchy — SRAM (Flash Attention tiles), HBM (active KV cache), CPU DRAM (evicted cache) — that mirrors classical memory hierarchy design in computer architecture.

Figure 5: GPU compute utilization across attention methods and sequence lengths on H100. FlashAttention-3 with FP8 maintains the highest utilization across all sequence lengths, while standard attention drops below 10% beyond 16K tokens due to memory bottlenecks.

4.4 Implications for Resource-Constrained Deployment #

The mobile and edge deployment scenario analyzed by recent work on I/O caching for resource-constrained platforms (Zhang et al., 2025^[14]) reveals that Flash Attention’s memory benefits become even more critical when GPU HBM is limited. On devices with 8-16 GB of memory, the quadratic attention term would limit context to approximately 2K tokens without Flash Attention; with it, 32K-token contexts become feasible when combined with GQA and INT4 quantization.

The RingX scalable parallel attention framework (Fang et al., 2025^[15]) demonstrates that Flash Attention’s tiling principle extends to distributed multi-GPU settings, enabling ring-based sequence parallelism where each GPU processes a tile of the attention computation and communicates partial results via point-to-point messaging. This bridges kernel-level and system-level memory optimization.

The emerging field of vectorized Flash Attention for RISC-V architectures (Kim et al., 2026^[16]) and hardware-optimized exponential computation via FLASH-D (Lee et al., 2025^[17]) suggests that Flash Attention’s principles are being embedded into hardware design itself, moving from a software optimization to a hardware-software co-design paradigm.

graph TB
    subgraph Hardware
        SRAM[GPU SRAM 20MB\nFlash Attention tiles]
        HBM[GPU HBM 80GB\nActive KV cache]
        CPU[CPU DRAM 512GB\nEvicted KV cache]
    end
    subgraph Optimizations
        FA[Flash Attention\nO N memory]
        GQA2[GQA\n4-8x head reduction]
        QUANT[INT4 Quantization\n4x compression]
        PRUNE[Token Pruning\n2-5x reduction]
    end
    SRAM --> |tiles| HBM
    HBM --> |eviction| CPU
    CPU --> |recompute| HBM
    FA --> SRAM
    GQA2 --> HBM
    QUANT --> HBM
    PRUNE --> HBM

5. Conclusion #

This article investigated Flash Attention’s role as the foundational kernel technology for memory-efficient LLM inference, answering three research questions through analysis of 2025-2026 research spanning ACL, NAACL, AAAI, EACL, NeurIPS, and systems venues.

RQ1 Finding: Flash Attention reduces attention peak memory from O(N squared) to O(N) by eliminating materialization of the full attention matrix through IO-aware tiling. Measured by Peak HBM Usage Ratio (PHUR) = 1.02 for Flash Attention versus 1024 for standard attention at 128K sequence length (d=128). This matters for our series because it establishes the kernel-level memory floor upon which all higher-level cache optimizations (pruning, sharing, compression) operate.

RQ2 Finding: Flash Attention variants are asymmetrically effective across inference phases: FlashAttention-3 excels at prefill (740 TFLOPs/s, 75% GPU utilization in FP16 on H100) while FlashInfer provides superior decode performance (29-69% ITL reduction). Measured by Time-to-First-Token and Inter-Token Latency across batch sizes 1-32. This matters for our series because the prefill-decode asymmetry determines which memory optimizations are most impactful at each inference stage, directly connecting to our earlier analysis of cache-aware scheduling.

RQ3 Finding: Flash Attention enables a multiplicative memory reduction pipeline when combined with GQA (8x), INT4 quantization (4x), and selective pruning (2.5x), achieving 97.5% total KV-cache reduction from 32 GB to 0.8 GB for a 70B model at 4096 context with less than 2% quality degradation. Measured by Memory Compression Ratio (MCR) = 2.5%. This matters for our series because it demonstrates that the AI Memory optimization techniques we have studied across 18 articles compose effectively, with Flash Attention as the enabling foundation.

The next articles in this series will examine distributed KV-cache management across multi-GPU serving systems and disaggregated prefill-decode architectures — topics where Flash Attention’s tiling principle extends from single-GPU kernels to cluster-scale memory coordination.

Reproducibility: All analysis code and data are available at github.com/stabilarity/hub/tree/master/research/ai-memory/.