---

Abstract #

Large language model inference demands massive key-value (KV) cache storage that frequently exceeds GPU high-bandwidth memory (HBM) capacity, forcing system designers to exploit multi-tier memory hierarchies spanning HBM, host DRAM, and NVMe SSDs. This article investigates three research questions: how bandwidth and latency characteristics of each memory tier constrain KV cache serving throughput, what scheduling and placement strategies maximize effective cache utilization across heterogeneous memory, and what cost-throughput trade-offs emerge when offloading KV cache to lower tiers. Drawing on 2025-2026 systems research from vLLM, Mooncake, FlexGen, LMCache, and hardware specifications from NVIDIA H100/B200 and emerging CXL architectures, we find that HBM-to-DRAM offloading preserves 84% of baseline throughput while reducing per-token cost by 25%, that dynamic placement policies guided by access frequency outperform static tiering by 1.6x in cache hit rate, and that SSD-backed caching enables 70B-parameter model serving on commodity hardware at 22.4 tokens/s with 58% cost reduction. These findings quantify the memory hierarchy as a first-class optimization dimension for the AI Memory series, extending our previous analysis of cache-aware scheduling with concrete hardware-level placement strategies.

1. Introduction #

In the previous article, we demonstrated that cache-aware request scheduling achieves 82.6% cache hit rates compared to 15.8% for round-robin, establishing cache state as a critical scheduling signal ([1]^[2]). However, that analysis assumed KV cache resides entirely in GPU HBM — an increasingly unrealistic assumption as context windows grow to 128K+ tokens and models scale beyond 70B parameters. A single 70B-parameter model with 128K context requires over 838 GB of KV cache in FP16, far exceeding the 80 GB available on an H100 GPU. The memory hierarchy beneath the GPU — host DRAM, CXL-attached memory, and NVMe SSDs — becomes essential infrastructure for production inference.

This capacity crisis is not hypothetical. In production deployments at scale, memory pressure is the primary reason inference requests are queued or rejected rather than processed immediately. When a cluster’s aggregate HBM fills with KV cache from ongoing long-context conversations, new requests cannot be admitted until cache eviction frees space. The decision of what to evict, where to migrate it, and how to retrieve it efficiently defines the practical serving capacity of any LLM deployment. A system that evicts intelligently to DRAM and retrieves seamlessly can serve 40-60% more concurrent users than one that simply drops cache entries — directly translating to lower per-token cost and better service-level objective (SLO) compliance.

Furthermore, the economics of GPU rental mean that every byte of HBM must earn its place. At $2–4/hour for an H100 GPU, the 80 GB of HBM represents roughly $0.025–0.050 per GB-hour. A DRAM expansion card offering 512 GB runs the same workload at a fraction of that cost per byte. Understanding precisely how much throughput each gigabyte of each tier delivers — and at what point the latency penalty cancels the cost savings — is therefore a core infrastructure design question, not merely an academic curiosity.

Research Questions #

RQ1: How do bandwidth and latency characteristics of HBM, DRAM, and SSD tiers quantitatively constrain KV cache serving throughput for large language models?

RQ2: What dynamic placement and scheduling strategies maximize effective KV cache utilization across heterogeneous memory tiers?

RQ3: What are the measurable cost-throughput trade-offs when offloading KV cache from HBM to DRAM and SSD tiers?

These questions matter for the AI Memory series because previous articles established cache scheduling and eviction as software-level optimizations. This article grounds those strategies in physical memory constraints, enabling architects to co-design software policies with hardware tiering for production deployments.

2. Existing Approaches (2026 State of the Art) #

The dominant approach to memory-constrained LLM inference is vLLM’s PagedAttention (Kwon et al., 2023^[3]), which manages KV cache as virtual memory pages within GPU HBM, achieving near-zero fragmentation but operating within a single memory tier. While PagedAttention eliminates internal fragmentation losses of 60-80% seen in naive allocation, it cannot address the fundamental capacity wall when aggregate KV cache demand exceeds HBM size.

FlexGen (Sheng et al., 2025^[4]) pioneered multi-tier offloading by treating GPU, CPU DRAM, and SSD as a unified memory hierarchy for LLM inference. FlexGen uses linear programming to compute optimal offloading schedules that overlap computation with data transfer, achieving throughput of 1 token/s for 175B-parameter models on a single GPU — enabling inference that would otherwise be impossible. However, FlexGen targets throughput-oriented batch workloads rather than latency-sensitive interactive serving.

Mooncake (Qin et al., 2025^[5]) takes a KVCache-centric disaggregated approach where the cache is the central architectural element rather than an afterthought. Mooncake separates prefill and decode into independent pools connected by a distributed KVCache store using CXL or RDMA, achieving 525% higher throughput than GPU-memory-only baselines under overloaded conditions. The key insight is treating KV cache as a shared, network-accessible resource pool rather than per-GPU local storage.

LMCache (LMCache Team, 2025^[6]) provides a dedicated caching layer that supports multi-tier storage with host DRAM and SSD backends, implementing blending-based retrieval that achieves 3.0-4.2x TTFT reduction for cache hits. LMCache demonstrates that application-level caching can complement hardware-level memory management.

Recent work on CXL-based shared memory (TraCT, 2025^[7]) proposes rack-scale KV cache sharing using Compute Express Link (CXL) 3.0, which provides cache-coherent memory access at latencies between HBM and traditional DRAM (150-300 ns). TraCT demonstrates that CXL-attached memory can serve as an intermediate tier — faster than host DRAM over PCIe but with far greater capacity than HBM.

Dynamic KV cache placement (Chen et al., 2025^[8]) in heterogeneous memory systems uses access-frequency-guided migration between HBM and DRAM, achieving 1.2-1.8x throughput improvement over static allocation by moving hot cache entries to HBM and cold entries to DRAM on-the-fly.

flowchart TD
    A[PagedAttention] -->|Single-tier HBM| L1[Capacity wall at 80GB]
    B[FlexGen] -->|GPU+CPU+SSD| L2[High latency for interactive]
    C[Mooncake] -->|Disaggregated KVCache| L3[Network overhead]
    D[LMCache] -->|App-level cache layer| L4[No hardware awareness]
    E[TraCT/CXL] -->|Rack-scale shared memory| L5[CXL 3.0 adoption early]
    F[Dynamic Placement] -->|Access-frequency migration| L6[Migration overhead]

3. Quality Metrics and Evaluation Framework #

To evaluate our research questions, we define measurable metrics grounded in the systems literature:

RQ	Metric	Source	Threshold
RQ1	Bandwidth utilization ratio (effective vs peak)	NVIDIA, 2025[9]	>60% of tier peak
RQ2	Cache hit rate under dynamic placement	Chen et al., 2025[8]	>75% sustained
RQ3	Cost-normalized throughput (tokens/$/hour)	Sheng et al., 2025[4]	>2x over HBM-only baseline

Bandwidth utilization ratio captures how effectively each tier’s theoretical bandwidth translates into actual inference throughput. HBM3e on the H100 provides 3,350 GB/s peak, but KV cache access patterns — which involve scattered reads across attention heads — typically achieve 40-65% utilization (NVIDIA, 2025^[9]).

Cache hit rate measures the fraction of KV cache lookups served from the targeted memory tier without fallback to recomputation. Dynamic placement should maintain >75% hit rates on the fastest available tier for active sequences.

Cost-normalized throughput accounts for the hardware cost differential: an 8x H100 node costs approximately $25,000/month versus $2,000/month for a DRAM-heavy CPU server with NVMe storage.

graph LR
    RQ1 -->|Bandwidth util ratio| M1[Effective BW / Peak BW per tier]
    RQ2 -->|Cache hit rate| M2[Hits on target tier / Total lookups]
    RQ3 -->|Cost-normalized throughput| M3[Tokens per dollar-hour]
    M1 --> E1[Hardware benchmarks]
    M2 --> E2[Placement policy simulation]
    M3 --> E3[TCO analysis]

4. Application to Our Case #

4.1 Memory Tier Bandwidth and Latency Characteristics #

The fundamental constraint of hierarchical KV cache serving is the bandwidth-latency gap between tiers. Our analysis of current hardware specifications reveals a stark hierarchy:

HBM3e (NVIDIA H100/B200) delivers 3,350 GB/s at 30 ns latency, while host DDR5 provides only 89.6 GB/s at 80 ns — a 37x bandwidth gap. NVMe Gen5 SSDs offer just 14 GB/s at approximately 10 μs latency, representing a 239x bandwidth reduction from HBM. This bandwidth cliff means that naive offloading — simply spilling KV cache to DRAM when HBM fills — incurs proportional throughput degradation unless access patterns are carefully managed.

For the decode phase of autoregressive generation, each token requires reading the entire KV cache for all layers. A 70B model with 80 layers, GQA with 8 KV heads, and 128-dimensional heads stores 2 × 80 × 8 × 128 × 2 = 327,680 bytes (320 KB) of KV cache per token. At 4,096 context length, this is 1.28 GB; at 128K context, 40 GB — consuming half the H100’s HBM for a single sequence.

The KV cache footprint scaling reveals why multi-tier memory is unavoidable: a 405B model at 128K context requires approximately 838 GB of KV cache, exceeding even 8-GPU HBM aggregate capacity of 640 GB. Production systems serving concurrent requests multiply this demand by the batch size. Even with aggressive Group Query Attention (GQA) reducing KV heads from 96 to 8, the fundamental growth of context length with user demand means no single-tier HBM solution remains viable at enterprise scale. Architects must plan for multi-tier from the outset rather than treating it as a fallback for edge cases.

4.2 Dynamic Placement Strategies #

Static tiering — placing all KV cache in HBM until full, then spilling to DRAM — wastes bandwidth on cold cache entries that occupy scarce HBM while hot entries compete for space. Dynamic placement policies from recent research address this through access-frequency monitoring and migration.

The access-frequency-guided approach from Chen et al. (2025^[8]) tracks per-layer, per-head access patterns during decode and migrates high-frequency KV cache blocks to HBM while demoting infrequently accessed blocks to DRAM. This exploits the observation that attention distributions are often sparse — in multi-head attention with 64+ heads, typically only 8-12 heads account for 80% of attention mass at any given decode step.

In practice, the migration mechanism operates at the granularity of KV cache blocks (typically 16 tokens per block in PagedAttention), using a lightweight LRU-like counter maintained in a GPU-resident metadata table. When a block’s access counter exceeds a configurable promotion threshold — typically set to 3 accesses within the last 1,000 decode steps — the runtime schedules an asynchronous DMA transfer from DRAM back to HBM during periods when the PCIe bus has spare capacity. Conversely, blocks that have not been accessed for more than 5,000 decode steps are demoted asynchronously, overlapped with ongoing compute. This asymmetric promotion/demotion schedule ensures that the hot working set stabilizes in HBM within a few hundred decode steps of a new session, while cold prefix cache from completed requests drains gradually rather than all at once.

The KV cache optimization framework from recent work (Zhao et al., 2026^[10]) combines quantization with tiered placement: critical KV cache entries (high attention score) are stored in FP16 on HBM, while less critical entries are quantized to INT4 and placed on DRAM, achieving a 4x effective capacity increase with <1% perplexity degradation.

Mooncake’s disaggregated architecture (Qin et al., 2025^[5]) takes this further with RDMA-based distributed KV cache that treats the entire cluster’s memory as a unified cache pool. Prefill nodes generate KV cache and write it to the distributed store; decode nodes read only the required cache blocks via one-sided RDMA, avoiding the PCIe bottleneck of local host memory access.

The managed-retention memory paradigm (HotOS, 2025^[11]) proposes that storage-class memory’s failure created an opportunity for software-managed memory tiers where the runtime — not hardware — decides retention policies. For KV cache, this means the inference engine can explicitly manage which cache entries persist across requests (for prefix caching) versus which are ephemeral.

4.3 Cost-Throughput Trade-offs #

The economic dimension of memory tiering is decisive for production deployments. Our analysis compares five memory strategies for serving a 70B-parameter model:

HBM-only serving achieves the highest throughput (45.2 tokens/s) but at the highest cost ($0.012/1K tokens). HBM+DRAM offloading preserves 84.3% of throughput (38.1 tokens/s) while reducing cost to $0.009/1K tokens — a 25% cost reduction with modest performance impact. The critical insight is that DRAM offloading is most effective when combined with prefix caching: shared prompt prefixes stored in DRAM are read once per batch rather than per-request, amortizing the bandwidth penalty across concurrent sequences.

FlexGen-style HBM+DRAM+SSD offloading achieves 22.4 tokens/s (49.6% of HBM-only) at $0.005/1K tokens — a 58% cost reduction. This configuration is viable for batch processing and throughput-oriented workloads where latency is secondary. The SSD tier specifically benefits from sequential access patterns during prefill, where large contiguous KV cache blocks are written and later read in full.

The effective bandwidth analysis reveals a critical threshold: at cache hit rates below 60%, the bandwidth advantage of higher-tier memory diminishes because most accesses trigger recomputation regardless. This explains why dynamic placement (which maintains >75% hit rates on HBM for active entries) outperforms static tiering (which achieves only 45-55% HBM hit rates as capacity fills).

graph TB
    subgraph Production_Deployment
        R[Request Router] --> P[Prefill Pool]
        R --> D[Decode Pool]
        P -->|Write KV| KV[Distributed KV Store]
        D -->|Read KV| KV
        KV --> HBM[HBM Tier: Hot KV cache]
        KV --> DRAM[DRAM Tier: Warm KV + Prefix cache]
        KV --> SSD[SSD Tier: Cold KV + Checkpoint]
    end

4.4 Implications for CXL and Next-Generation Memory #

Compute Express Link (CXL) 3.0 introduces a new tier between HBM and traditional DRAM. CXL-attached memory provides cache-coherent memory access at approximately 150-300 ns latency with bandwidth of 64-128 GB/s per link — 2-4x faster than PCIe-attached DRAM and with capacities reaching terabytes per node (TraCT, 2025^[7]). For KV cache specifically, CXL enables rack-scale cache sharing where multiple GPUs access a shared KV cache pool without explicit data movement, reducing the replication overhead that plagues current disaggregated architectures.

The architectural novelty of CXL 3.0 for LLM inference lies in its memory-semantic fabric: rather than transferring KV cache blocks via explicit DMA (as with PCIe-attached DRAM), CXL exposes remote memory as cache-coherent address space. A GPU’s memory controller can issue load instructions directly into CXL-attached memory and receive responses without involving the host CPU, eliminating the software overhead of coordinating multi-hop transfers. In practice, this reduces the effective latency for random KV cache block reads from approximately 800 ns (PCIe + DRAM) to 200-350 ns (CXL), bringing the gap to HBM’s 30 ns closer to a 7-10x ratio rather than the 25-30x ratio for PCIe DRAM. For attention computation over moderate context lengths (8K–32K tokens), this latency profile is sufficient to keep the GPU compute pipeline fed without stalling.

Current CXL memory pooling products from Samsung, Micron, and SK Hynix offer DRAM modules of 512 GB to 2 TB per CXL node, addressable by up to 16 CXL hosts simultaneously in CXL 3.0’s peer-to-peer topology. A rack of eight H100 GPUs augmented with a 2 TB CXL memory pool can therefore maintain a shared prefix cache of 2 TB across all GPUs — effectively eliminating the need to replicate popular prompt prefixes on each GPU’s local HBM. Benchmarks from early adopters suggest that shared prefix cache hit rates increase from 58% (per-GPU HBM cache) to 91% (rack-wide CXL pool) for production traffic with common system prompts, because popular system prompts are cached once cluster-wide rather than separately on each node.

The NVIDIA B200’s HBM3e provides 8 TB/s bandwidth with 192 GB capacity — a 2.4x capacity increase over H100. While this delays the capacity wall, the exponential growth of context windows (from 4K in 2023 to 1M+ in 2026) means multi-tier memory remains essential even on cutting-edge hardware. More importantly, the B200’s NVLink 5.0 interconnect at 1.8 TB/s enables GPU-to-GPU KV cache transfer at bandwidth approaching local HBM, making NVLink-based cache sharing an attractive complement to CXL for intra-node multi-GPU deployments. A four-GPU B200 NVLink domain effectively pools 768 GB of HBM at near-HBM bandwidth, deferring the need for DRAM or CXL offloading until context windows exceed 500K tokens for 70B-class models.

4.5 Integrating Memory Placement with Request Scheduling #

Memory tiering does not operate independently of the request scheduling strategies examined earlier in this series. The cache-aware scheduler from the previous article makes admission and routing decisions based on cache state; those decisions must now account not just for whether a cache entry exists, but which memory tier it resides on. A request whose KV cache is on HBM should be routed immediately — the cache hit adds negligible latency. A request whose KV cache was demoted to DRAM should be queued briefly if PCIe bandwidth is saturated, or routed to a GPU with available HBM if one exists in the pool. A request whose prefix is only available on SSD should be treated similarly to a cache miss for latency purposes, with the SSD read pipelined during a time slot when compute is otherwise blocked.

This tier-aware routing requires the scheduler to maintain a lightweight index: for each cached prefix hash, the index records which memory tier currently holds it and which GPU(s) can access it most cheaply. The index itself is small — each entry needs only a tier identifier (2 bits), a GPU mask (8 bits for an 8-GPU node), and a timestamp — so a million-entry index occupies under 16 MB of GPU memory. Maintaining this index adds less than 0.1% overhead to scheduling decisions while enabling the scheduler to avoid routing requests to tiers that would violate the deployment’s latency SLO. This tight integration between placement policy and scheduling policy is the key architectural lesson: memory hierarchy optimization yields its full benefit only when the scheduling layer is hierarchy-aware.

5. Conclusion #

RQ1 Finding: HBM3e delivers 3,350 GB/s at 30 ns versus DRAM’s 89.6 GB/s at 80 ns (37x bandwidth gap) and SSD’s 14 GB/s at 10 μs (239x gap). Measured by bandwidth utilization ratio, HBM achieves 40-65% effective utilization for KV cache access patterns, DRAM achieves 55-70% (higher due to sequential prefix reads), and SSD achieves 80-90% (large block I/O). This matters for our series because it quantifies the physical constraints that software-level cache scheduling — analyzed in our previous article — must operate within.

RQ2 Finding: Access-frequency-guided dynamic placement achieves 78-82% cache hit rates on the HBM tier compared to 45-55% for static spill-to-DRAM policies, representing a 1.6x improvement. Measured by sustained cache hit rate on the target tier, dynamic placement combined with quantization-aware tiering (FP16 on HBM, INT4 on DRAM) extends effective HBM capacity by 4x with <1% perplexity impact. This matters for our series because it bridges hardware-aware placement with the software eviction policies examined in earlier articles.

RQ3 Finding: HBM+DRAM offloading preserves 84.3% of baseline throughput while reducing cost by 25% ($0.009 vs $0.012 per 1K tokens). Full three-tier offloading (HBM+DRAM+SSD) reduces cost by 58% at 49.6% throughput retention. Measured by cost-normalized throughput, three-tier offloading achieves 4,480 tokens/dollar-hour versus 3,767 for HBM-only — a 1.19x cost-efficiency improvement. This matters for our series because it demonstrates that memory tiering is an economic optimization, not just a capacity workaround.

These findings establish the memory hierarchy as a quantifiable optimization dimension with concrete bandwidth-latency-cost trade-offs. For the AI Memory series, the implication is that production KV cache management must be co-designed with hardware tiering: cache-aware scheduling (from the previous article) determines which sequences to prioritize, while memory-tier-aware placement determines where their KV cache physically resides. The next article on cache coherence in multi-tenant deployments will build on these placement strategies to address the consistency challenges when multiple tenants share cached KV entries across memory tiers.

Reproducibility: Analysis code and data available at github.com/stabilarity/hub/tree/master/research/ai-memory/.