---

Abstract #

Compressing the key-value (KV) cache is the most direct lever for cutting the cost of long-context inference, but published results usually report memory savings or benchmark scores in isolation. We ask what compression buys in a production serving unit and what it costs in quality. We built a deterministic model from public numbers: attention configurations of five open-weight models, HBM capacity and bandwidth of NVIDIA H100 and H200 GPUs, on-demand GPU prices read on 21 October 2026, vLLM’s default memory settings, and the realised compression ratios and LongBench scores of six methods measured under one harness. Three results stand out. First, at 32K tokens a single H100 holds 13 concurrent Llama-3.1-8B sequences with a BF16 cache, 26 with FP8 and 70 with 2-bit KIVI; at 128K, Qwen3-32B on one H200 fits a single BF16 sequence. Second, once memory binds (above roughly 1,000 to 9,000 tokens), decode cost per million output tokens falls almost exactly by 1 – 1/c for compression ratio c: from $1.8297 to $0.3471 (81.0%) for Llama-3.1-8B at 32K, and by more when the uncompressed cache barely fits. Third, quality loss is method-specific, not ratio-specific: KIVI keeps at least 94.0% of the uncompressed score on all 12 model-task pairs, while TurboQuant and CaM keep only 59.5% to 64.7% on summarisation, which gives KIVI 13.55 to 13.8 percentage points of cost saved per point of worst-case quality lost against 1.82 to 1.97 for TurboQuant and CaM.

1. Introduction #

Our previous article in this series, Interpretable Models vs Post-Hoc Explanations: True Cost Comparison for Enterprise AI[1]^[2], argued that the cheapest option on paper is often not the cheapest in operation once hidden costs are counted. KV-cache compression is a sharp example of the same problem in inference. Every token a transformer has read is kept in GPU memory as key and value vectors for every layer, so the cache grows linearly with context length and with concurrency. For long prompts it, not the weights, decides how many requests one GPU can serve.

A large literature promises relief. Quantization stores the cache in fewer bits, eviction drops tokens that attention is unlikely to revisit, and architectural changes such as multi-head latent attention (MLA) shrink the cache at the source. The hub’s earlier benchmark review compared these families on accuracy at matched compression ratios[2]^[3]. What that comparison cannot tell an operator is the price effect: how many more users fit, what a million output tokens then costs, and whether the quality loss is acceptable for the specific workload. A 2025 study of production serving found that compression algorithms are still not widely adopted in production, partly because throughput gains shrink once FlashAttention and PagedAttention are in place and partly because compressed models can produce longer outputs[3]^[4]. A 2026 benchmark that ran four representative methods under a single harness concluded that the compression ratio alone is a poor predictor of end-to-end performance[4]^[5].

This article joins the two with an explicit cost model that uses realised, not nominal, compression ratios and pairs each method with its measured quality on the same tasks.

RQ1: How much KV-cache memory does each production model configuration need per token, and how many concurrent 8K, 32K and 128K sequences fit on one serving unit at BF16, FP8, 4-bit and 2-bit precision? RQ2: What do these capacity gains imply for decode cost per million output tokens, and when does the saving depart from the compression ratio? RQ3: How much task quality do representative methods give up at their realised compression ratio on LongBench workloads, and which methods sit on the cost-quality frontier?

The answers matter for the Cost-Effective Enterprise AI series because self-hosted inference is increasingly sold on a cost-per-token basis, and the KV cache is the term in that cost that grows with the length of the work.

2. Existing Approaches (2025–2026 state of the art) #

2.1 Low-precision formats in serving engines #

The lowest-risk option is to store the cache in a smaller floating-point format. vLLM supports FP8 KV caches in E4M3 and E5M2 formats with per-tensor or per-head scales, and its documentation notes that without calibration all scales default to 1.0 and that some layers, such as sliding-window attention, are more sensitive to KV-cache quantization[5]^[6]. The engine’s current option list goes well beyond FP8: it includes per-token-head INT8 and INT4, NVFP4, several TurboQuant variants and a 4-bit UltraQuant format[6]^[7]. NVIDIA reports that its NVFP4 KV cache reduces the cache footprint by about 50% relative to FP8 with an accuracy loss of less than 1% against BF16 and FP8 baselines on LiveCodeBench, MMLU-PRO, MBPP and RULER at 64K[7]^[8]. The current NVFP4 implementation dequantizes to FP8 before the attention math, so the saving is in capacity and bandwidth, not in arithmetic[7]^[8].

2.2 Asymmetric low-bit quantization #

Below 4 bits, the structure of the cached values starts to matter. KIVI observed that keys should be quantized per channel and values per token, keeps a small window of recent tokens in full precision, and reports 2.6 times less peak memory, up to 4 times larger batches and 2.35 to 3.47 times higher throughput on real workloads[8]^[9]. Rotation-based schemes such as TurboQuant take a data-oblivious route: a random rotation spreads outliers before uniform scalar quantization. For context-heavy multi-round workloads on AMD GPUs, UltraQuant, a 4-bit design anchored on TurboQuant-style quantization, sustained 2.71 times (MiniMax-M2.5) and 4.38 times (Qwen3-235B) the qualified-request throughput of a BF16 cache on replayed production coding-assistant traces, matching or exceeding hardware FP8 while using half its KV bytes[9]^[10]. Where it fails, low-bit quantization fails quietly: across eleven instruction-tuned models, Mistral-7B lost 15.2% of its refusals at only 1.03 times baseline perplexity, with no universal safe bit-width[10]^[11].

2.3 Token eviction and pruning #

Eviction keeps only the entries that future queries are expected to need. SnapKV selects them from an observation window at the end of the prompt[11]^[12]. KVzip scores each entry by how well the model can reconstruct the original context from the cache, which makes the compressed cache reusable across different queries; it reports 3 to 4 times smaller caches and about 2 times lower FlashAttention decoding latency with negligible loss, and shows that query-aware methods degrade even at a 90% cache budget in multi-query settings[12]^[13]. Its successor KVzap works during both prefill and decoding and reaches 2 to 4 times compression with negligible loss on Qwen3-8B, Llama-3.1-8B and Qwen3-32B[13]^[14]. The KVPress library behind these methods packages more than 20 techniques[14]^[15]. The risk of eviction is irreversibility. Under multi-instruction prompting, certain instructions degrade much faster than others as compression rises, and eviction bias can leak system prompts[15]^[16]. SCBench, which tests caches shared across turns, found that methods with sub-linear memory suffer in multi-turn use[16]^[17].

2.4 Architectural compression #

Changing the attention design shrinks the cache at the source. DeepSeek-V3 caches a 512-dimensional latent plus a 64-dimensional decoupled positional key per layer instead of full per-head keys and values[17]^[18]. Conversion methods bring this to existing models: MHA2MLA cuts the Llama2-7B KV cache by 92.19% with a 0.5% drop on LongBench while recovering performance with only 0.3% to 0.6% of the data[18]^[19], and TransMLA compresses 93% of the Llama-2-7B cache for a 10.6 times speedup at 8K context, needing 6 billion tokens of fine-tuning to recover quality[19]^[20].

2.5 Offloading and cache reuse #

The last family moves the cache rather than shrinking it. LMCache stores KV caches outside GPU memory and shares them across engines and queries, reporting up to 15 times higher throughput with vLLM on multi-round question answering and document analysis; its operators also observed that context truncation can halve the prefix-cache hit ratio[20]^[21]. EVICPRESS jointly chooses compression and eviction across storage tiers and reports up to 2.19 times faster time-to-first-token at equal quality[21]^[22]. On public cloud instances, a selection framework found that KV offloading saved up to 20.19% for offline workloads, while for online workloads the bigger gain (up to 73.7%) came from choosing cheaper instances without offloading[22]^[23]. Commercial APIs expose the same economics as a discount: Anthropic bills a cache hit at 10% of the base input price on most models[23]^[24].

flowchart TD
    A[KV-cache cost levers 2026] --> B[Fewer bits per entry]
    A --> C[Fewer entries]
    A --> D[Smaller entries by design]
    A --> E[Cheaper memory tier]
    B --> B1[FP8 and NVFP4 in serving engines]
    B --> B2[KIVI, TurboQuant, UltraQuant]
    C --> C1[SnapKV, KVzip, KVzap]
    C --> C2[Merging such as CaM]
    D --> D1[MLA, MHA2MLA, TransMLA]
    E --> E1[LMCache, EVICPRESS, provider prompt caching]

3. Method and Data #

Model configurations. We took layer counts, KV-head counts and head dimensions from the Llama 3 report for Llama-3.1-8B and the 70B architecture shared by Llama-3.3-70B[24]^[25], from the Qwen3 report for Qwen3-32B (64 layers, 64 query and 8 KV heads)[25]^[26], and from the DeepSeek-V3 report for its MLA dimensions[17]^[18]. Head dimensions and exact parameter counts come from each model’s public Hugging Face configuration and weight metadata, recorded in data/models.csv. For grouped-query attention, the cache holds 2 × layers × KV heads × head dimension values per token; for MLA it holds layers × (latent + positional) values.

Hardware and prices. An H100 SXM has 80 GB of HBM at 3.35 TB/s[26]^[27] and an H200 has 141 GB at 4.8 TB/s[27]^[28]. We priced them at RunPod’s on-demand Secure Cloud rates read on 21 October 2026: $3.99 per hour for an H100 SXM and $5.29 for an H200[28]^[29]. We define three serving units with BF16 weights: Llama-3.1-8B on one H100, Qwen3-32B on one H200, and Llama-3.3-70B on two H200s with tensor parallelism.

Capacity and cost model (our estimate). The memory available for the cache is 92% of HBM, vLLM’s default --gpu-memory-utilization[6]^[7], minus the weights. That leaves 57.54 GB for the 8B unit, 64.2 GB for Qwen3-32B and 118.33 GB for the 70B unit. Concurrency is the number of whole sequences of a given length that fit, capped at 256 (our assumption for a scheduler limit). Decode is treated as bandwidth-bound: each step reads the weights once and the cache of every active sequence, and step time is total bytes divided by aggregate bandwidth. Cost per million output tokens is the hourly price divided by tokens per hour. The model ignores dequantization compute, scale metadata, activations and prefill, so its savings are an upper bound; Section 4.2 compares them with measured results.

Compression ratios and quality. For FP8 we use the nominal 2 times and for NVFP4 the nominal 4 times implied by NVIDIA’s “about 50% of FP8”[7]^[8]. For six methods we use realised ratios and LongBench scores from Agrawal and Mayer, who ran KIVI at 2 and 4 bits, TurboQuant at 3 and 4 bits, SnapKV at a 0.75 compression setting and the merging method CaM on Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 on one A100 40GB under one harness[4]^[5]. Their Table 4 gives realised compression of 5.16 to 5.32 for KIVI-2, 3.15 to 3.19 for KIVI-4, 4.27 for TurboQuant-3, 3.76 for TurboQuant-4, 4.00 for SnapKV and 1.00 to 13.82 for CaM; we use the mean over context buckets. Quality retention is each method’s score divided by the uncompressed score, per dataset, over six datasets (HotpotQA, 2WikiMQA, Qasper, MultiFieldQA-en, TriviaQA, MultiNews) and two models.

All code, data and outputs are in stabilarity/hub/research/kv-cache-compression-techniques-in-production-cost-savings-vs-quality-degradation-trade-offs.

flowchart LR
    M[Model configs: layers, KV heads, head dim] --> K[KV bytes per token]
    G[GPU specs and prices] --> C[Capacity model]
    K --> C
    F[Format ratios: nominal and realised] --> C
    C --> R1[RQ1 concurrency]
    C --> T[Bandwidth-bound decode]
    T --> R2[RQ2 cost per 1M output tokens]
    Q[LongBench scores, one harness] --> R3[RQ3 cost-quality frontier]
    R2 --> R3

4. Results #

4.1 RQ1: The cache, not the model, sets concurrency beyond a few thousand tokens #

With a BF16 cache, Llama-3.1-8B and Mistral-7B each need 128 KiB per token, Qwen3-32B needs 256 KiB and Llama-3.3-70B 320 KiB. One 128K-token sequence therefore occupies 17.18 GB on the two small models, 34.36 GB on Qwen3-32B and 42.95 GB on the 70B model. DeepSeek-V3’s MLA cache needs 68.62 KiB per token, or 9.21 GB per 128K sequence, which is 4.66 times less than Llama-3.3-70B despite a far larger model.

Table 1 turns these sizes into concurrency.

Serving unit	Context (tokens)	BF16	FP8 (2x)	KIVI-4 (3.17x)	NVFP4 or SnapKV (4x)	KIVI-2 (5.24x)
Llama-3.1-8B, 1x H100	8,192	53	107	169	214	256
Llama-3.1-8B, 1x H100	32,768	13	26	42	53	70
Llama-3.1-8B, 1x H100	131,072	3	6	10	13	17
Qwen3-32B, 1x H200	32,768	7	14	23	29	39
Qwen3-32B, 1x H200	131,072	1	3	5	7	9
Llama-3.3-70B, 2x H200	32,768	11	22	34	44	57
Llama-3.3-70B, 2x H200	131,072	2	5	8	11	14

Maximum concurrent sequences. Source: model configurations[24]^[25][25]^[26], GPU specifications[26]^[27][27]^[28], realised ratios[4]^[5]; our calculation, capped at 256.

Chart 2 shows the regime change. Below 1,715 tokens the BF16 batch on the 8B unit is limited by the scheduler cap, so compression adds nothing; above it, every doubling of context halves concurrency and each format shifts the line up by its ratio. With 2-bit KIVI, the memory limit begins only at 8,986 tokens. For Qwen3-32B on one H200, the BF16 threshold is 957 tokens, because the weights leave just 64.2 GB for the cache. For retrieval, document analysis and multi-turn sessions, the KV format therefore decides how many users a GPU serves.

4.2 RQ2: The saving tracks 1 – 1/c, with a bonus at the edge of memory #

When the batch is memory-limited, each decode step reads roughly the same bytes, the weights plus a full cache budget, whatever the format. Compression does not make a step faster; it lets the step serve more sequences. Cost per token therefore falls in proportion to concurrency, and the saving converges to 1 – 1/c whatever share of the bytes the cache represents. In our model the cache is 78% of the bytes read per step on the 8B unit and 46% to 49% on the larger units, yet the savings are almost identical.

Serving unit, context	BF16 (USD/1M)	FP8 (USD/1M)	KIVI-4 (USD/1M)	4x formats (USD/1M)	KIVI-2 (USD/1M)	KIVI-2 saving
Llama-3.1-8B, 8K	0.4555	0.2273	0.1435	0.1136	0.0886	80.6%
Llama-3.1-8B, 32K	1.8297	0.9149	0.5748	0.4555	0.3471	81.0%
Llama-3.1-8B, 128K	7.4551	3.7275	2.3244	1.8297	1.3973	81.3%
Qwen3-32B, 32K	5.4953	2.7476	1.7017	1.3491	1.0162	81.5%
Qwen3-32B, 128K	30.5779	11.9458	7.33	5.4953	4.2362	86.2%
Llama-3.3-70B, 32K	7.2142	3.6071	2.3075	1.8035	1.3852	80.8%
Llama-3.3-70B, 128K	34.7473	15.2137	9.5475	7.2142	5.5948	83.9%

Decode cost per 1M output tokens, idealised bandwidth-bound model. Source: GPU specifications[26]^[27][27]^[28]; RunPod on-demand prices accessed 2026-10-21[28]^[29]; our calculation.

At 32K tokens on the 8B unit, FP8 halves the cost from $1.8297 to $0.9149 per million output tokens, KIVI-4 saves 68.6%, the 4-times formats save 75.1% and KIVI-2 saves 81.0%, against a theoretical 80.9% for a ratio of 5.24. The deviations come from integer batches. At 128K tokens, Qwen3-32B fits only one BF16 sequence in its 64.2 GB budget but three in FP8, so FP8 saves 60.9% instead of 50%, and Llama-3.3-70B gains 56.2% from FP8. Compression pays most precisely where the uncompressed cache barely fits. Below the cap threshold the saving shrinks: on the 8B unit at 8K tokens, KIVI-2 is capped at 256 sequences and saves 80.6%.

Two caveats apply. Cost per output token still rises almost linearly with context, by a factor of about 16 between 8K and 128K on the 8B unit, so compression lowers the curve without flattening it. And the model is an upper bound; real kernels deliver less. Gao and colleagues found that compression methods often cannot achieve more than a 1.5 times increase in decoding throughput, and that more than 20% of ShareGPT samples produced responses at least 1.5 times longer under compression, which can cancel the speedup end to end[3]^[4]. In the single-harness benchmark, SnapKV matched or exceeded uncompressed throughput, KIVI and CaM showed modest throughput loss and TurboQuant a severe drop[4]^[5]. That benchmark ran one A100 40GB and does not report a batch-size sweep, so it does not show the memory-saturated regime our model describes, which is the regime in which KIVI’s original 4 times larger batches and 2.35 to 3.47 times throughput were obtained[8]^[9]. A practical reading is that realised savings fall between the measured values at low concurrency and 1 – 1/c at full memory.

4.3 RQ3: Quality loss depends on the method and the task, not on the ratio #

Chart 4 places the six methods on cost and quality axes.

Method	Realised compression	Cost at 32K (USD/1M)	Saving vs BF16	Mean retention	Worst retention (pair)	Saving per worst-case point lost
KIVI-2	5.24x (5.16–5.32)	0.3471	81.0%	100.6%	94.0% (Mistral, HotpotQA)	13.55
KIVI-4	3.17x (3.15–3.19)	0.5748	68.6%	102.3%	95.0% (Mistral, HotpotQA)	13.8
SnapKV-0.75	4.00x	0.4555	75.1%	99.6%	84.9% (Llama, MultiNews)	4.96
CaM	3.57x (1.00–13.82)	0.5111	72.1%	97.0%	63.4% (Mistral, MultiNews)	1.97
TurboQuant-3	4.27x	0.426	76.7%	90.6%	60.0% (Mistral, MultiNews)	1.92
TurboQuant-4	3.76x	0.4842	73.5%	93.0%	59.5% (Llama, MultiNews)	1.82

Source: realised compression and LongBench scores from Agrawal and Mayer, Tables 2 and 4[4]^[5]; cost for Llama-3.1-8B on one H100 at 32K tokens, our calculation. Retention above 100% reflects run-to-run variation and task effects, not a gain to rely on.

Three findings follow. First, the ratio does not predict the damage. TurboQuant-4 compresses less than SnapKV (3.76 against 4.00 times) yet loses far more on summarisation, keeping 59.5% of the MultiNews score on Llama against SnapKV’s 84.9%. KIVI-2 compresses most and keeps 99.4% on the same task. Second, summarisation is the stress test: every method except KIVI loses at least 8% there on at least one model, and TurboQuant and CaM lose 35% to 41%, while question answering and few-shot tasks barely move. The benchmark authors attribute this to small, systematic reconstruction errors on all tokens accumulating in tasks that need global context, and note that their TurboQuant summarisation runs were executed once because each took three to four days[4]^[5]. Third, among the compressed options only KIVI-2 and KIVI-4 sit on the cost-quality frontier on both mean and worst-case retention, and they buy 13.55 and 13.8 percentage points of cost saving per point of worst-case retention lost, roughly seven times the rate of TurboQuant and CaM and almost three times that of SnapKV.

CaM shows a second production risk: its realised compression ranged from 1.00 on NarrativeQA to 13.82 on GovReport at over 8K tokens, so the capacity it frees cannot be planned in advance[4]^[5]. A plan built on its average ratio of 3.57 would be wrong for most individual workloads.

The FP8 and NVFP4 formats are absent from this harness. NVIDIA’s results put NVFP4 within 1% of BF16 on its four benchmarks[7]^[8], but a 2026 cost-quality study of Qwen2.5-7B on vLLM found that a naive FP8 KV cache kept normal throughput while answering none of 200 GSM8K questions correctly[29]^[30]. The same study found FP8 weights retained 99.4% of baseline accuracy. The contrast points at calibration and kernel support, not the bit-width, as the deciding factor, which is consistent with vLLM’s warning that uncalibrated scales default to 1.0[5]^[6].

5. Discussion #

Implications for practitioners. Treat the KV format as a capacity decision. For workloads that hold fewer than about 1,000 to 2,000 tokens per sequence on these units, the scheduler cap binds first and compression changes little; beyond about 9,000 tokens even a 5-times format is memory-bound, and the saving approaches 1 – 1/c. FP8 is the default first step because it is built into vLLM and halves decode cost at long context, but it must be validated on the target model with calibrated scales, given the GSM8K collapse reported for a naive configuration[29]^[30]. KIVI-style asymmetric quantization is the strongest choice when workloads are unknown, because it is the only family here that never fell below 94.0% of the uncompressed score. Eviction methods such as SnapKV are attractive for question answering and few-shot tasks, where they matched or beat the full cache, but summarisation, multi-turn sessions[16]^[17] and long system prompts[15]^[16] need separate testing. Methods with variable compression should not underpin capacity plans.

flowchart TD
    S[Tokens held per sequence] --> T{Above memory threshold, about 2K to 9K?}
    T -->|No| N[Batch cap binds, compression has little value]
    T -->|Yes| U{Workload known and narrow?}
    U -->|No| K[KIVI-style or calibrated FP8 first]
    U -->|Yes| V{Needs global context, e.g. summaries?}
    V -->|Yes| K
    V -->|No| E[Eviction such as SnapKV or KVzip, test multi-turn]
    K --> X[Validate safety and instruction following]
    E --> X
    X --> P[Plan capacity on realised, not nominal, ratio]

The hidden quality dimensions. LongBench averages are only part of the risk. KV quantization can remove refusals without moving perplexity, as the 15.2% refusal loss on Mistral-7B shows[10]^[11], and eviction can drop late instructions or leak system prompts[15]^[16]. KVDiagnosis, a 2026 diagnostic benchmark, starts from the observation that aggregate task scores reveal neither which correct answers fail under compression nor why[30]^[31]. For enterprise deployments, a compression rollout therefore needs a regression suite that includes instruction-following and safety prompts, not only task accuracy.

Self-hosting versus buying cache discounts. Provider prompt caching prices reused input at 10% of the base rate on most Anthropic models[23]^[24], and reuse systems such as LMCache cut repeated prefill[20]^[21]; neither lowers the decode-side capacity cost measured here, so the levers are complementary. Architectural compression remains the most powerful lever per token. MLA’s 4.66-times smaller cache than Llama-3.3-70B’s grouped-query layout is comparable to a 4-bit quantizer before any quantizer is applied, and conversions such as MHA2MLA report a 0.5% LongBench drop for a 92.19% cache cut[18]^[19], but they need fine-tuning and are not deployment switches.

Limitations and threats to validity. The cost model is idealised: it ignores dequantization compute, quantization scales and residual windows beyond what the realised ratios include, activation memory, prefill cost and longer outputs, and assumes 100% of peak bandwidth. A realistic bandwidth efficiency would raise absolute dollar values but not the relative savings, since it scales all formats alike. Prices are one provider’s on-demand rates on one day. The quality data come from one harness, two 7B to 8B models, six datasets and an A100 40GB[4]^[5]; larger models may tolerate compression differently, and single runs on summarisation leave room for noise. Nominal ratios for FP8 and NVFP4 are not measured on the same tasks.

What would change the conclusion. A kernel stack that makes compressed attention slower per byte than BF16 at high batch would cut the realised saving below 1 – 1/c. A harness showing TurboQuant-class methods at above 95% on summarisation with calibrated rotations would move them onto the frontier. Evidence that KIVI loses safety alignment on a given model[10]^[11] would override its quality lead for that deployment.

6. Conclusion #

RQ1 Finding: Beyond a few thousand tokens per sequence, KV-cache size, not model size, sets how many requests a GPU serves. Measured by maximum concurrent sequences at 32K tokens for Llama-3.1-8B on one H100 = 13 with BF16, 26 with FP8 and 70 with 2-bit KIVI, and by the memory-bound threshold = 1,715 tokens for BF16 on that unit. This matters for the series because capacity planning for long-context services must start from the KV format, not from the parameter count.

RQ2 Finding: When memory binds, decode cost falls by almost exactly 1 – 1/c, and by more when the uncompressed cache barely fits. Measured by cost per million output tokens for Llama-3.1-8B at 32K = $1.8297 with BF16 and $0.3471 with KIVI-2, a saving of 81.0%, and by the FP8 saving for Qwen3-32B at 128K = 60.9% against a nominal 50%. This matters for the series because it gives a simple, auditable rule for pricing compression before any benchmark is run, while measured kernels and longer outputs set the lower bound.

RQ3 Finding: Quality loss is a property of the method and the task, not of the compression ratio. Measured by worst-case LongBench retention = 94.0% for KIVI-2 against 59.5% to 63.4% for TurboQuant and CaM, and by cost saved per point of worst-case retention lost = 13.55 to 13.8 for KIVI against 1.82 to 1.97 for TurboQuant and CaM. This matters for the series because a cheaper cache is only cost-effective if it fails gracefully on the workloads it actually serves.

The next question for the series is how the self-hosted cost per long-context token compares with API list prices once realistic bandwidth efficiency and prefill are included.