KV, Prefix, Prompt and Semantic Caching in LLMs Explained
This tutorial breaks down the four cache layers in an LLM serving stack from first principles: the KV cache for a single request, prefix caching for cross-request reuse on the server, provider-billed prompt caching, and a semantic cache that returns stored responses by embedding similarity. It includes runnable code for transformers DynamicCache/StaticCache, a compact vLLM-style chain-hash block scheduler, an Anthropic prompt-caching example, and a tiny semantic cache. Concrete numbers ground each trade-off: a 70B model holds roughly 40GB of KV tensors at 128K context; Anthropic bills reads at 0.1x and writes at 1.25x input rates; two sentences differing only by a negation score 0.952 cosine similarity yet need opposite answers. It also covers what silently breaks reuse: variable content in system prompts, reordered RAG chunks, partial tail blocks, eviction, and per-tenant salt separation. The piece is honest about failure modes, including a throughput regression on unique traffic and the false-positive risk of semantic caching. Useful for engineers operating inference, tuning RAG cost, or debugging prompt-cache misses.