---

Abstract #

Token costs are the largest variable expense in production AI systems. For enterprises running thousands of daily API calls, optimising how context is stored, reused, and compressed is not an architectural nicety — it is the difference between a viable product and an unscalable one. This article provides a practitioner’s map of the three caching layers now available to enterprise AI teams — KV-cache reuse via provider prompt caching, application-layer semantic caching, and prompt compression — and explains how to combine them to achieve 60–80% cost reductions without sacrificing response quality. The techniques described here require no model changes and are deployable today against any major provider.

1. The Token Cost Problem at Scale #

An enterprise chatbot handling 50,000 queries per day against a 10,000-token system prompt, at Claude Sonnet input pricing, costs roughly $2,500/day in input tokens alone — $900,000 per year — before any output tokens are counted. At GPT-4o pricing the figure is similar. At frontier model scale (Claude 3.7 Opus, GPT-4.5), it is higher still.

The arithmetic is unforgiving: every token sent to the API is billed, every time, unless the infrastructure prevents it. Most enterprises do not yet have infrastructure that prevents it.

Ivchenko (2026), Inference Economics^[2] documented that falling per-token prices have not translated into falling total inference bills, because usage growth outpaces price reduction. The logical response is not to wait for cheaper models but to send fewer tokens in the first place.

Three techniques, applied in combination, attack this problem at different layers:

1. Provider-side KV caching — reusing prefix computations already on the GPU
2. Application-layer semantic caching — serving stored responses when queries are semantically equivalent
3. Prompt compression — reducing the token count of context before it reaches the API

Each addresses a different fraction of the total token spend. The optimal strategy layers all three.

---

2. Provider Prompt Caching: KV Reuse at the API Level #

2.1 How It Works #

Large language models process text in two phases: prefill (computing key-value attention matrices for the input) and decode (generating output tokens one at a time). Prefill is computationally expensive and proportional to input length. KV caching stores the prefill output — the attention matrices — so that subsequent requests reusing the same prefix skip the computation entirely.

Both Anthropic and OpenAI now expose this at the API level. Anthropic’s prompt caching documentation (2026)^[3] describes two modes: automatic caching (enabled by default on Claude 3.5+ models) and explicit cache breakpoints using cache_control markers on specific content blocks. The pricing multiplier for cache reads is 0.1× the base input rate — 90% cheaper. Cache writes cost 1.25× base, with a minimum cached prefix of 1,024 tokens and a TTL of 5 minutes (extendable to 1 hour via explicit breakpoints).

OpenAI’s prompt caching (2026)^[4] operates similarly: cached tokens are billed at 50% of the standard input rate on GPT-4o and o-series models, with automatic cache hit detection for prefixes of 1,024+ tokens. No code changes are required; the discount applies transparently.

sequenceDiagram
    participant App as Application
    participant API as LLM API
    participant GPU as GPU KV Store

    App->>API: Request 1 (system prompt + query A)
    API->>GPU: Prefill [system prompt] — MISS — compute & store
    GPU-->>API: KV cache written (10,000 tokens)
    API-->>App: Response A

    App->>API: Request 2 (system prompt + query B)
    API->>GPU: Prefill [system prompt] — HIT — retrieve cache
    GPU-->>API: KV cache retrieved (0 compute)
    API-->>App: Response B [90% cheaper input]

2.2 What Gets Cached and Why Prefix Stability Matters #

Cache hits depend on the request sharing an identical prefix with a previous cached request. This means:

• System prompts are the highest-value caching target: they are identical across all requests from the same application.
• Few-shot examples embedded in the system block are equally cacheable.
• Retrieved document chunks (RAG results) are cacheable if they are stable across requests (e.g., a fixed knowledge base loaded at startup), but not if they vary per query.
• Conversation history is partially cacheable: the growing prefix of prior turns is reusable, but the current user message is always new.

The practical implication is that system prompt architecture deserves careful attention. Anything stable — persona instructions, tool schemas, fixed reference text — should be front-loaded into a single contiguous prefix block. Anything variable (retrieved context, current query) should be appended after the cache breakpoint.

flowchart LR
    subgraph Cacheable["Cacheable Prefix (stable)"]
        A[System persona\n~500 tokens]
        B[Tool schemas\n~1,500 tokens]
        C[Fixed knowledge base\n~8,000 tokens]
    end
    subgraph Variable["Variable Suffix (not cached)"]
        D[RAG results\n~2,000 tokens]
        E[Conversation history\n~1,000 tokens]
        F[User query\n~100 tokens]
    end
    Cacheable --> Variable
    style Cacheable fill:#d4edda,stroke:#28a745
    style Variable fill:#fff3cd,stroke:#ffc107

2.3 Realistic Cost Projections #

For the 50,000-query/day chatbot example above, if 80% of input tokens are system prompt (cacheable) and the cache hit rate is 90%:

Token category	Daily tokens	Cost without cache	Cost with cache
System prompt (cached reads)	400M	$2,000	$200
System prompt (cache writes)	44M	$220	$275
Variable suffix	50M	$250	$250
Total input	—	$2,470	$725

That is a 71% input cost reduction with no quality loss. The numbers shift somewhat per provider and model, but the order of magnitude holds.

---

3. Application-Layer Semantic Caching #

3.1 The Semantic Cache Concept #

Provider caching eliminates redundant prefill computation for identical prefixes. It does not help when two requests ask the same question in different words. Semantic caching operates at a higher level: it stores (query, response) pairs and, when a new query arrives, checks whether a semantically equivalent query has been answered before. If yes, the stored response is returned without an API call.

Redis (2024), What is Semantic Caching^[5] describes the architecture: queries are embedded using a lightweight embedding model, stored in a vector database with their corresponding responses, and retrieved via approximate nearest-neighbour search. Requests within a configurable cosine similarity threshold (typically 0.90–0.95) are treated as cache hits.

GPTCache (Zilliz, 2024)^[6] is the most-used open-source implementation, integrating with LangChain and LlamaIndex. It supports multiple similarity backends (cosine, Euclidean, inner product) and multiple storage backends (Redis, PostgreSQL, MongoDB, SQLite). For enterprise deployments, Redis with vector search or Milvus are the typical production choices.

3.2 When Semantic Caching Pays #

Semantic caching is most effective for:

• FAQ-style chatbots: where a large fraction of queries fall into a small set of semantic clusters (customer support, HR helpdesks, internal knowledge assistants)
• Code generation assistants: where developers repeatedly ask for similar boilerplate patterns
• Report summarisation pipelines: where the same document is summarised for multiple stakeholders

It is least effective for:

• Real-time analysis requiring fresh data (market prices, live dashboards)
• Personalised content where responses must differ by user
• Creative tasks where variation is a feature, not a bug

flowchart TD
    Q[User Query] --> E[Embed query\n~0.1ms, ~$0.00001]
    E --> S{Similarity search\nvector DB}
    S -->|Hit ≥ 0.92| C[Return cached response\n~1ms, $0]
    S -->|Miss| A[LLM API call\n~800ms, $0.02-0.10]
    A --> W[Write to cache]
    W --> R[Return response]
    C --> R
    style C fill:#d4edda,stroke:#28a745
    style A fill:#f8d7da,stroke:#dc3545

3.3 Cache Hit Rates in Practice #

Cache hit rates depend heavily on query diversity. Enterprise workloads typically cluster into recognisable patterns:

• Internal HR/policy chatbots: 40–60% hit rate (employees ask the same questions repeatedly)
• Customer support agents: 30–50% hit rate after the first week of operation
• Developer tooling: 20–40% hit rate (code patterns recur)
• Open-ended research assistants: 5–15% hit rate (queries are too diverse)

At a 40% cache hit rate, eliminating those API calls reduces total cost by approximately 40%. Combined with provider prompt caching on the remaining 60% of calls, total cost reduction reaches 60–75%.

---

4. Prompt Compression #

4.1 The Compression Problem #

Prompt caching reduces the cost of reusing context. Compression reduces the token count of context that cannot be cached — retrieved documents, conversation history summaries, user-provided files.

Jiang et al. (2023), LLMLingua^[7] introduced a budget-controlled iterative compression algorithm using a small auxiliary language model (LLaMA-7B class) to score token-level perplexity and remove low-information tokens. On standard benchmarks, LLMLingua achieves 4–20× compression with less than 5% performance degradation at conservative compression ratios (2–4×).

Jiang et al. (2024), LongLLMLingua^[8] extended this to long-context scenarios, adding question-aware saliency scoring — tokens more relevant to the query are preserved; irrelevant tokens are aggressively pruned. This is particularly useful for RAG pipelines where retrieved chunks often contain substantial irrelevant text.

4.2 Practical Compression Strategies #

For enterprise systems, three compression strategies address different cost buckets:

Conversation history compression: Instead of appending full message history, periodically summarise older turns into a compact state representation. At 10,000 tokens of history, a 4× compression produces a 2,500-token summary — a 7,500-token saving per request.

RAG context compression: Retrieved chunks average 400–600 tokens each. At 5 chunks per query, that is 2,000–3,000 tokens of retrieved context. Question-aware compression (LongLLMLingua-style) can reduce this to 500–800 tokens — a 70% reduction.

System prompt compression: For system prompts too large to fit efficiently in cache, LLMLingua can reduce a 12,000-token system prompt to 4,000–5,000 tokens with minimal instruction-following degradation.

Context Component	Before Compression (tokens)	After Compression (tokens)	Reduction
Conversation history	10,000	2,500	75%
RAG chunks	3,000	800	73%
System prompt (cached)	12,000	0 variable cost	90%+
Total variable tokens	~25,000	~3,500	86%

4.3 Compression Quality Trade-offs #

Compression introduces latency (the auxiliary model must process the prompt) and quality risk. The economics of compression depend on:

• Token savings: At 3× compression on 20,000-token prompts, savings per call are roughly $0.04–0.08 at current frontier pricing.
• Compression compute cost: A 7B auxiliary model on a single GPU processes ~1,000 tokens/second; a 20,000-token prompt takes ~20 seconds. For synchronous pipelines, this may exceed the LLM call latency itself.
• Viable use cases: Compression is most economical for batch/async pipelines (nightly report generation, document ingestion), not interactive chatbots.

For interactive applications, simpler heuristics — sliding window history (keep last N turns), extractive summarisation, fixed-length retrieval limits — achieve 40–60% of the savings at near-zero latency cost.

---

5. Layered Strategy: Combining All Three Techniques #

The maximum cost reduction comes from applying all three techniques non-redundantly. They operate on different token categories and compound rather than overlap:

Technique	Target tokens	Typical reduction	Applies to
Provider KV caching	System prompt + stable context	70–90%	All providers
Semantic caching	Full API calls (duplicate queries)	30–60% of calls eliminated	Application layer
Prompt compression	Variable context (RAG, history)	50–80% of variable tokens	Remaining calls

A worked example for a RAG-based enterprise search agent (100,000 daily queries):

Baseline	Per query tokens	Daily cost
System prompt (10,000 tokens × 100,000)	1B input tokens	$5,000
RAG context (3,000 tokens × 100,000)	300M input tokens	$1,500
Output (500 tokens × 100,000)	50M output tokens	$750
Total baseline	—	$7,250/day

After optimisation:

• Provider caching on system prompt: 90% reduction → -$4,500/day
• Semantic caching: 40% cache hit rate eliminates 40% of all remaining calls → -$350/day
• RAG compression at 3×: remaining 60% of calls use 1,000-token context → -$600/day
• Net daily cost: ~$1,800 (75% reduction)
• Annual saving: ~$2,000,000

Ivchenko (2026), Agent Cost Optimization as First-Class Architecture^[9] argues that this kind of infrastructure investment should be designed in at the architecture stage, not retrofitted. The compounding nature of these optimisations means the savings scale with volume; for a system handling 1M queries/day, the annual saving from the same stack exceeds $20M.

---

6. Implementation Roadmap #

Organisations should implement in order of effort-to-impact ratio:

Week 1 — Provider caching (zero code change) Enable cache breakpoints in system prompts. Move all stable content (persona, tools, fixed knowledge) to the cacheable prefix. Measure cache hit rate via provider dashboards.

Month 1 — Semantic caching layer Deploy a vector store (Redis, Milvus, or Weaviate). Instrument the application to compute query embeddings and check for cache hits before API calls. Start with conservative similarity thresholds (0.95) and lower gradually as quality is confirmed.

Month 2–3 — Context management discipline Implement conversation history summarisation at configurable turn thresholds. Apply fixed-length RAG context limits. Measure token-per-query reduction.

Month 3–6 — Compression pipelines (for batch workloads) Deploy LLMLingua or LongLLMLingua for batch document processing and overnight report generation. Measure cost savings against compression latency overhead.

---

7. Measurement Framework #

Cost optimisation is meaningless without measurement. The following metrics should be instrumented:

• Cache hit rate (provider): available in API dashboards; target >80% for system prompts
• Cache hit rate (semantic): log in application; target >30% after 1 week of operation
• Average tokens per request (input + output, tracked weekly)
• Cost per query (total API spend / total queries)
• Quality regression rate: sample 1% of semantic cache hits for human or LLM-as-judge evaluation; threshold at <1% degradation

Ivchenko (2026), The Meta-Meta-Analysis^[10] documented that measurement methodology matters more than the specific metric chosen. Token cost reduction without quality measurement is not optimisation — it is cost-cutting with unknown side effects. The measurement framework above treats both dimensions equally.

---

Conclusion #

The 60–80% cost reduction claimed in this article’s title is not an edge case or a marketing figure. It is achievable — and has been achieved by engineering teams who have implemented provider caching, semantic caching, and prompt compression in combination. The techniques are available today, require no model changes, and operate transparently to end users.

The investment required is modest: provider caching requires a system prompt refactor; semantic caching requires a vector store and ~200 lines of middleware; compression requires a small auxiliary model or simpler heuristics for interactive use cases. For any system processing more than 10,000 queries per day, the payback period is measured in weeks.

Token economics, like compute economics before them, reward the engineers who measure carefully and build deliberately. The teams who instrument these optimisations now will have a durable cost advantage over those who wait for providers to lower prices further — because their competitors will be running the same optimisation stack against lower prices, maintaining the same structural advantage.