KV Cache Is Becoming the Core Object of AI Infrastructure

For most of the last decade, AI infrastructure was designed around one verb: training. Clusters were built for dense matrix multiplication, benchmarked in FLOPs, and sized by how many GPUs you could fit in a rack. Inference was the afterthought — the thing you did once the expensive part was over.

That assumption expired this year. Gartner's August 2026 forecast puts worldwide AI-optimized IaaS spending at $42.3 billion, up 96.4% year over year, and reports that for the first time inference spending ($23.3 billion) has surpassed training ($19 billion). Inference is 55% of that market in 2026 and is forecast to reach 59% in 2027. The crossover is structural, not a rounding event: agentic systems turn every user request into dozens of model calls, and unlike a training run, inference never stops.

Here is the part that matters for anyone building on top of these systems. Training is compute-bound. Serving is state-bound. And the state that makes serving state-bound has a name: the KV cache. What began as an internal optimization — cache the key and value tensors so you don't recompute them every token — has quietly become the object around which inference economics, serving architecture, and now storage hardware are being reorganized.

The memory math that started it

The KV cache scales with a formula that is trivial to write and brutal to live with: 2 × layers × KV heads × head_dim × sequence length × batch size × bytes per element. The PagedAttention paper (Kwon et al., SOSP 2023) gave the canonical example: for a 13B-parameter OPT model, a single token's KV cache costs 800 KB (2 × 5120 hidden size × 40 layers × 2 bytes), meaning one request at 2048 tokens could consume 1.6 GB. On a 40 GB A100 serving that model, roughly 65% of memory went to static weights and close to 30% to KV cache — and of that KV cache memory, only 20.4% to 38.2% actually held token state. The rest was lost to fragmentation and over-allocation.

Apply the same formula to a Llama-3-70B-class model in fp16 (80 layers, 8 KV heads, head_dim 128) and you get roughly 320 KB per token — which is why a 1M-token context implies a KV cache on the order of 300 GB, several times the model's own weights. Long context is not primarily a compute problem. It is a memory-capacity problem, and the memory in question is the KV cache.

Three awakenings

First, it became a memory-management problem. PagedAttention borrowed virtual memory and paging from operating systems, treating KV cache blocks as pages, tokens as bytes, and requests as processes. The result was near-zero waste and block-level sharing across requests, delivering 2–4× throughput over FasterTransformer and Orca at the same latency. This reframed the KV cache as something to be allocated, shared, and evicted — a resource, not a tensor.

Second, it became a priced commodity. vLLM shipped automatic prefix caching in v0.4.0, and the commercial API providers followed with something more consequential than a performance feature: a discount. OpenAI's prompt caching documentation shows the discount deepening as the inference stack matured — 50% for GPT-4o, 75% for GPT-4.1, and 90% for the GPT-5 family ($0.125 vs $1.25 per million input tokens), with up to ~80% latency reduction and 67% faster time-to-first-token at 150k+ token prompts. Anthropic's published multipliers are explicit: cache writes at 1.25× base input (5-minute TTL) or 2× (1-hour), and cache reads at 0.1× — Claude Sonnet 4.6 drops from $3 to $0.30 per million tokens on a hit. DeepSeek went further and put its cache on disk, launching context caching with cache hits at ¥0.1 per million tokens against ¥1 for misses.

A cache hit is now a 90%-off coupon you have to engineer and then defend. OpenAI's own guidance is blunt on this: the prefix must be stable, prompts under 1024 tokens never cache at all, and reordering your tool definitions silently busts everything downstream.

Third, it became a scheduling object. The clearest expression is Mooncake, the platform behind Moonshot AI's Kimi, which won the Erik Riedel Best Paper Award at USENIX FAST 2025 under the title "Mooncake: Trading More Storage for Less Computation." Mooncake separates prefill and decoding into independent clusters and pools the idle CPU, DRAM, SSD, and NIC capacity of the GPU cluster into a distributed KV cache (Mooncake Store), orchestrated by a KV-cache-aware global scheduler. The measured results: on real workloads, Kimi handles 75% more requests; effective request capacity improves 59%–498%; A800 and H800 clusters serve 115% and 107% more requests than the previous system; thousands of nodes process over 100 billion tokens daily. Its RDMA Transfer Engine moves KV cache at 87 GB/s over 4×200 Gbps and 190 GB/s over 8×400 Gbps — 2.4× and 4.6× faster than TCP. Global cache hit rates run up to 136% higher than node-local caching, cutting prefill GPU time by as much as 48%.

The open-source LMCache project pushes the same idea further, treating KV cache as something you store, compress, search, move, and observe across GPU, CPU DRAM, local disk, and remote backends. Its academic ancestor, CacheGen (SIGCOMM 2024), showed KV cache can be encoded 3.5–4.3× smaller with a 3.2–3.7× TTFT reduction; AMD's January 2026 benchmarks with vLLM on MI300X report 3–10× improvements on long-document QA.

2026: it moves into the hardware

The strongest signal that this is not a software fad came at CES in January 2026, when NVIDIA announced the Inference Context Memory Storage (ICMS) Platform, powered by the BlueField-4 DPU. The press release states the thesis outright: "A KV cache cannot be stored on GPUs long term, as this would create a bottleneck for real-time inference in multi-agent systems." ICMS extends GPU memory capacity and enables high-bandwidth KV cache sharing across rack-scale clusters, claiming up to 5× tokens per second and up to 5× power efficiency compared with traditional storage, integrated with DOCA, NIXL, and Dynamo over Spectrum-X Ethernet. Twelve storage vendors — Dell, HPE, IBM, Pure Storage, VAST Data, WEKA, DDN, Hitachi Vantara, Nutanix, Supermicro, Cloudian, and AIC — signed on, with availability in the second half of 2026. Jensen Huang's framing was unusually direct: "AI is revolutionizing the entire computing stack — and now, storage."

Hardware is being reshaped around the same object. Rubin CPX, announced in September 2025, is billed as the first CUDA GPU purpose-built for massive-context processing: 30 PFLOPS of NVFP4 compute, 128 GB of GDDR7, and 3× the attention performance of GB300 NVL72, with the Vera Rubin NVL144 CPX rack packing 8 exaflops and 100 TB of fast memory. And in August 2026, SK hynix and Sandisk published the first open HBF (High Bandwidth Flash) specification through the Open Compute Project — a new tier between HBM and SSD offering up to 512 GB per stack at 0.4–3.0 TB/s over UCIe, with Google and Tenstorrent participating in validation.

A storage tier, a GPU category, and a memory standard have all been created for one data structure.

The counter-force: architectures trying to delete it

Running in the opposite direction is a decade of architectural work aimed at making the KV cache smaller or unnecessary. Multi-head Latent Attention (DeepSeek-V2, 2024) compresses KV into a latent vector, cutting cache size by 93.3% and raising maximum generation throughput 5.76×. Native Sparse Attention (DeepSeek, 2025) reports 11.6× decoding speedup at 64K sequences; DeepSeek Sparse Attention in V3.2 lowered API prices by 50–75%, with costs at 128K roughly 3.5× lower for prefill and 6–7× lower for decoding. Kimi Linear (October 2025) uses a hybrid of Kimi Delta Attention and MLA to reduce KV cache usage by up to 75% while delivering up to 6× decoding throughput at 1M context — and, notably, beats full MLA under an identical training recipe.

None of this shrinks the KV cache in absolute terms, because every efficiency gain is immediately spent on longer contexts and more agents. Stanford's Digital Economy Lab found that agentic coding tasks consume up to 1,000× more tokens than simple code chat, that the same task can vary 30× in cost between runs, and that cost lands overwhelmingly on the input side — an agent re-reads its accumulated context before nearly every action. One 2026 audit of 30 production engineering teams, cited by Elastic, put re-sent context at 62% of the agent inference bill. Compression moves the frontier; it does not remove the category.

What builders should do about it

The practical consequence is that cache hit rate is now a first-class production metric, alongside latency and error rate. Concretely:

1. Instrument it. Every major API returns cached-token counts in its usage object. If you cannot see your hit rate, you cannot manage your largest cost lever.
2. Structure prompts for stable prefixes. Static content (system prompt, tool definitions, retrieved documents) first, variable content last. A single changed token turns discounted reads back into full-price ones.
3. Treat KV cache as a tiered storage problem, not a GPU problem. Systems like Dynamo, LMCache, and llm-d now manage it across HBM, host DRAM, local NVMe, and object storage; the right question is no longer "how much VRAM" but "how many tiers, and what hit rate does each hold."
4. Watch the standards layer. llm-d — a Kubernetes-native distributed inference stack donated to the CNCF Sandbox in March 2026 by IBM Research, Red Hat, and Google Cloud, with NVIDIA, CoreWeave, AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI as collaborators — makes KV-cache-aware routing and hierarchical offloading a control-plane concern. Its production data claims cache-aware routing cuts infrastructure cost 30–50% while holding latency SLOs, and AWS now ships a dedicated llm-d container with Elastic Fabric Adapter support. vLLM, meanwhile, has an open RFC on KV cache interoperability API standardization.

Here is the original insight worth stating plainly: the KV cache is undergoing the same transition databases underwent when the buffer pool stopped being an internal detail and became a managed, priced, tiered resource with its own vendors, formats, and standards bodies. The tell is not the papers — it is that NVIDIA built a DPU and a storage platform around it. You do not build a storage tier for an implementation detail. Whoever controls where the KV cache lives will control the cost of every token served.