Here's a line item hiding in almost every LLM bill: you're paying full price, over and over, to send the model the exact same text. Your system prompt. Your tool definitions. That 40-page policy document every request references. Prompt caching fixes this — providers store the repeated prefix of your prompt and charge you a fraction (in the best cases around a tenth) of the normal input rate every time it's reused. It is the single cheapest optimization in AI right now, and most teams still haven't turned it on.
This guide covers how prompt caching actually works, when cache hits happen (and the ways you silently break them), the break-even math, and how to restructure prompts so the discount applies to the biggest possible slice of your bill.
When a model processes your prompt, the expensive part is prefill — computing attention over every input token before generating the first output token. If the first N tokens of today's request are byte-identical to the first N tokens of a request from a minute ago, that computation is redundant. Prompt caching lets the provider keep the processed state of that prefix around and skip the recompute.
You see it as pricing: cached input tokens are billed at a steep discount to fresh input tokens — commonly 50–90% off depending on provider and model. Writing to the cache sometimes costs a premium over base input (a cache-write surcharge), which is why the math matters and why blindly caching everything can lose money on low-traffic prompts.
Caches match from the start of the prompt, forward, and stop at the first difference. This single fact should dictate your prompt architecture:
Put content in this order — most static at the top, most dynamic at the bottom:
1. System prompt and persona — never changes between requests.
2. Tool and function definitions — changes only when you ship.
3. Reference documents and few-shot examples — changes rarely.
4. Conversation history — grows, but the old turns stay identical.
5. The user's new message — changes every request, so it goes last.
Get this backwards — say, a timestamp or a request ID at the top of the system prompt — and every request misses cache from token one. A single dynamic word placed early can cost you the entire discount. This is the most common cache-buster we see, followed closely by non-deterministic serialization: if your tool definitions render in a different key order per request, the bytes differ, and the cache treats it as brand-new text.
Say your stable prefix is 20,000 tokens (system prompt + tools + a reference doc), and fresh input costs $3 per million tokens with cached input at $0.30 and a 25% cache-write premium.
Without caching, 1,000 requests process 20M prefix tokens at full rate: $60. With caching, you pay one write (20k tokens at $3.75/M ≈ $0.075) and 999 hits (about 20M tokens at $0.30/M ≈ $6). Total: roughly $6.08 — a 90% cut on the prefix portion of the bill. The catch is cache lifetime: entries typically expire after minutes of inactivity, so the discount depends on request frequency. A prompt hit once an hour re-writes every time and saves nothing. A prompt hit every few seconds rides the discount almost continuously. Before restructuring anything, run your own numbers in the free prompt caching calculator — it models write premiums, hit rates, and expiry against your actual traffic.
Prompt caching pays off in direct proportion to how much repeated context you carry. Agent systems are the extreme case: an agent loop re-sends its system prompt, tool schemas, and running history on every single step — ten tool calls means ten near-identical prefixes. Chatbots with long conversations, RAG systems with a fixed instruction preamble, and anything doing classification against a constant rubric all sit in the same bucket. Count what your prefix actually costs with the token counter, then check how much room it eats with the context window planner — the three numbers together tell you exactly what caching is worth to you.
Two honest caveats. First, caching discounts input, not output — if your costs are dominated by long generations, this lever barely moves the needle. Second, cache reads are still tokens the model attends over: latency improves (prefill is skipped), but a bloated 50k-token prompt is still a bloated prompt. Caching makes repetition cheap; it does not make bloat smart. Trim first, cache second.
Here's the part the API pricing pages won't tell you: prefix reuse isn't a billing feature, it's an inference feature. Run models on your own hardware and the same trick — prompt/prefill reuse in llama.cpp and friends — is simply free, forever, with no expiry window set by someone else's business model. That's the design principle behind QADIR OS: a sovereign agentic OS that runs local-first, where we measured prefill-reuse delivering roughly a 5× speedup on repeated prefixes on our own GPUs. No meter, no cache-write surcharge, no landlord.
Stop paying full price for the same tokens. Model your savings with the free prompt caching calculator, or skip the meter entirely — join QADIR OS early access and run agents on your own machine. No card required.