KV Cache, Prefix Caching, and KV-Aware Routing for LLM Inference

TL;DR: A KV cache stores attention state from tokens an LLM has already processed. Prefix caching reuses that state when later requests share the same beginning. KV-aware routing adds a placement decision: it sends the request to a replica that already holds the useful cache, balancing reuse against load.

Conversational AI repeatedly sends the same instructions, memories, and history back to a model. Recomputing that prefix on every turn increases prefill latency and input cost. Caching can remove the duplicate work, but only when the serving system recognizes the shared prefix and routes the request to the right memory.

What does a KV cache store?

Transformer attention converts earlier tokens into key and value vectors that later tokens use during generation. A KV cache preserves those vectors so the model does not recompute them for every output token or repeated request. It reduces work; it does not change the model’s learned weights.

vLLM implements automatic prefix caching with hashed blocks. A block’s identity includes its tokens, the parent-prefix hash, and additional values such as LoRA identifiers, multimodal hashes, or cache salts. Only full blocks are cached. When a new prompt shares those blocks, the scheduler can reuse them and compute only the unmatched suffix.

Cache memory remains finite. Blocks compete for GPU or host memory and are evicted when capacity is needed. The effective result depends on block size, prompt stability, eviction policy, concurrency, and how often later requests share a prefix.

How is prefix caching different from prompt caching?

Prompt caching is the broader product concept: a provider avoids reprocessing repeated prompt content and may bill cached input at a lower rate. Prefix caching is a technical method that reuses the longest matching beginning of a token sequence. The terms overlap, but they are not always identical across providers.

Some APIs make caching automatic; others require explicit cache controls or stable prompt boundaries. Inworld documents automatic caching for selected providers and explicit controls where the underlying provider requires them. The Inworld model catalog lists separate cache-read prices for many first-party and routed models.

Term	Primary meaning	Buyer question
KV cache	Stored attention state	How much memory and reuse are available?
Prefix caching	Reuse of matching initial token blocks	What content must remain stable?
Prompt caching	Provider feature and billing treatment	Is reuse automatic, explicit, or discounted?
KV-aware routing	Placement based on existing cache state	Will the next request reach the useful cache?

Why does routing determine cache value?

A cache hit can disappear when a load balancer sends a request to a replica that does not hold the matching blocks. Ordinary load balancing optimizes available capacity; KV-aware routing also considers which replica already holds the request’s prefix. The scheduler then trades cache locality against queue length and load.

Inworld’s October launch material describes a router that hashes prompt blocks, tracks replica holdings, and scores candidate replicas using prefix availability and load. In a replay of 5,000 production requests for GLM 5.3 Flash, the reported cache-hit rate increased from 25% to 47%. One dedicated customer deployment reportedly reached 80% in production.

These are first-party results and depend on the workload. The public page should include the prompt-length distribution, repeat interval, cache definition, replica count, eviction policy, model version, and raw replay output before presenting them as benchmark evidence.

How does caching change inference cost?

Caching reduces the work required to process repeated input and can lower the billed price of those tokens. The blended result depends on the share of tokens that hit the cache, the cached-input rate, and the number of repeated turns. A large discount does not create savings when the cache-hit rate is low.

blended input cost = uncached tokens × uncached rate + cached tokens × cache-read rate

The current Inworld model catalog lists separate cache-read prices for models including GLM 5.3 Flash, Gemma 4 26B, Gemma 4 31B, DeepSeek variants, Qwen, and MiMo. Those rates should be captured with the model ID and date because catalogs change.

Teams should measure cost per completed interaction rather than cache savings alone. Extra retries, poor routing, or lower-quality outputs can erase the token savings.

Which workloads benefit most?

Prefix caching is most valuable when requests share stable initial content. AI companions, tutors, roleplay applications, coding agents, document analysis, and support systems often reuse system instructions, memories, policies, tools, or long documents. Stable prefixes and short changing suffixes create the highest potential reuse.

It helps less when prompts are unrelated, prefixes change frequently, cache entries expire before reuse, or traffic is too dispersed to preserve locality. Large multimodal inputs require hashes that distinguish images or other embeddings, and multi-tenant systems need isolation controls to prevent information leakage through shared cache behavior.

vLLM supports cache salts that limit reuse to requests sharing the same salt. Security and privacy should be part of the cache design, not an afterthought added after optimizing hit rate.

How should teams test a cache system?

A useful cache test reports cold and warm behavior separately. It uses the application’s prompt distribution, measures matched tokens rather than request-level hits alone, and records where the cache lived. It also verifies that outputs remain correct and isolation works as intended.

Measure the repeated-prefix share in production traces.
Run cold requests before warming the cache.
Repeat identical and partially matching prompts.
Vary concurrency and replica count.
Record token-level reuse, TTFT, errors, and billed input.
Test eviction, spill, and tenant-isolation behavior.

Report both cache-hit rate and the change in completed-interaction cost. A high hit rate with negligible latency or cost improvement may indicate that the reused blocks are too small to matter.

Related Guides

Key Takeaways

A KV cache stores attention state so repeated tokens do not require the same computation again.
Prefix caching reuses matching initial blocks; prompt caching describes the broader provider feature and billing behavior.
KV-aware routing improves reuse only when cache locality outweighs queue and load costs.
Cache-hit rate should be paired with token reuse, latency change, and cost per completed interaction.
Multi-tenant cache systems require isolation, collision resistance, and explicit security testing.

Frequently Asked Questions

Does KV caching change model output?

Correct prefix-cache reuse does not change the model computation for the cached tokens; it reuses attention state that was already computed. Output can still differ because of sampling, model settings, provider versions, or an incorrect cache implementation. Teams should validate output quality and isolation alongside latency.

What is a good cache-hit rate?

There is no universal target. A conversational workload with stable system instructions and long history may support a high rate, while unrelated prompts may produce almost none. Measure the percentage of input tokens reused, not only requests with any hit, and connect it to latency and blended cost.

Is prefix caching safe in a multi-tenant service?

It can be, but the cache key and isolation design matter. vLLM documents cache salts that limit reuse to requests sharing the same salt and warns that non-cryptographic hashing can increase collision and privacy risk. Providers should disclose isolation behavior and test timing-based leakage scenarios.