Category: AI Memory

As large language models scale beyond the memory capacity of individual accelerators, distributing inference across multiple GPUs introduces fundamental challenges for key-value cache management. This article examines how tensor parallelism, pipeline parallelism, and emerging hybrid strategies partition KV-cache state across devices, analyzing the communication overhead, memory efficiency, and ...

Flash Attention has become the foundational kernel technology enabling memory-efficient inference in large language models (LLMs), transforming how attention computation interacts with GPU memory hierarchies. This article investigates three research questions: (1) how does Flash Attention's tiling strategy reduce peak memory consumption compared to standard attention, and what are the theoretic...

As large language models (LLMs) scale to context windows exceeding one million tokens, the key-value (KV) cache grows linearly and becomes the dominant memory bottleneck during autoregressive inference. Sliding window attention and compressive caching represent two complementary families of techniques that bound memory usage while preserving access to long-range context. This article investigat...

As large language models (LLMs) scale to billions of parameters and context windows stretch beyond 128K tokens, the key-value (KV) cache becomes the dominant memory bottleneck during inference. Cross-layer KV-cache sharing represents a family of techniques that exploit redundancy in key and value representations across transformer layers to reduce cache memory without retraining. This article i...

This article investigates token pruning and attention sparsity as complementary strategies for reducing KV-cache memory consumption during large language model inference. Building on our series analysis of semantic prompt caching, we examine how selective token removal and sparse attention patterns can achieve 50-80% memory reduction while preserving generation quality. Three research questions...

Prompt caching has emerged as a critical optimization for large language model (LLM) serving, yet production systems overwhelmingly rely on exact-match strategies that miss semantically equivalent queries. This article investigates semantic prompt caching — systems that identify and serve cached responses for semantically similar (but not identical) prompts using embedding-based similarity dete...

Speculative decoding has emerged as a transformative inference optimization that breaks the sequential bottleneck of autoregressive generation by drafting multiple tokens in parallel and verifying them in a single forward pass. This article examines three research questions at the intersection of speculative decoding and KV cache management: how draft-then-verify architectures interact with cac...

As large language models scale beyond hundreds of billions of parameters and context windows extend to millions of tokens, the key-value (KV) cache required for attention computation becomes the dominant memory bottleneck during inference. Grouped-Query Attention (GQA) addresses this by allowing multiple query heads to share fewer key-value heads, reducing cache footprint while preserving model...

As large language models scale to billions of parameters and millions of context tokens, the key-value (KV) cache that stores attention states becomes the dominant memory bottleneck during inference. Traditional contiguous memory allocation for KV caches leads to severe fragmentation — wasting 40-60% of available GPU memory — and fundamentally limits serving throughput. This article investigate...

The rapid expansion of context windows — from 4K tokens to 10M tokens in models like Llama 4 — has produced a proliferation of evaluation benchmarks, yet no unified framework exists for comparing long-context capabilities across these disparate tests. This article presents a meta-analysis of ten major context benchmarks (NIAH, RULER, LongBench v2, InfiniteBench, BABILong, NoLiMa, LongGenBench, ...