How Prompt Caching Works, Step by Step

In How Claude Code Works I showed that an agent re-reads the entire conversation on every turn. A session that adds 30k tokens of new content across eight turns bills for 100k+ input tokens, because the system prompt, tool definitions, and every prior tool result get sent again and again.

Prompt caching is what makes that affordable. The provider stores the work it already did on the repeated part of the prompt and charges a fraction of the price to reuse it, which is also why later turns in a long session are not much slower than early ones. It comes with some rules about how a prompt has to be built for the reuse to happen, and those rules are most of what this post is about.

Every major provider has a version of this. OpenAI applies it automatically to prompts over 1,024 tokens and discounts the cached portion. Anthropic makes you mark where the cacheable region ends. I’ll use Anthropic’s API for the concrete examples because the explicit markers make the mechanics visible, but the core rule is identical everywhere. This walkthrough builds it up in steps, and every diagram is interactive.

Step 1: Why it’s a prefix, not a lookup

Before a model can generate its first output token, it runs over every input token and computes an internal state for each one (in transformer terms, the attention keys and values). This phase is called prefill, and for a long prompt it is most of the input cost and most of the time before the first output token appears.

The state for token N depends only on tokens 1 through N. Nothing after it matters. So if two requests share the same first 40,000 tokens, the state for those 40,000 tokens is identical in both, and the second request can load it instead of recomputing it.

The moment the two requests differ, everything after the difference is different too, even if the later text happens to match. Prompt caching is a prefix match, checked from position zero forward.

Try “Put a timestamp at the top”. Two requests that are 95% identical share a prefix of zero tokens, because the very first tokens differ. Then try “Same question, different wording”, where the shared prefix runs almost to the end.

The match is exact at the byte level, so a reordered JSON key or an extra space counts as a difference:

The JSON case is the one that bites in real code. Anything you serialize into the prompt (a user profile, tool results, retrieved records) needs a stable key order, or the same data produces different bytes on different requests. Sort keys before serializing and normalize whitespace in one place, and do it at the point where the prompt is assembled rather than hoping every caller remembers.

The cached state also belongs to one specific model, since a different model would compute different state from the same tokens. Cache entries are also scoped to your organization; nobody else’s prompts can hit your cache and vice versa.

Step 2: What the prefix is in a real request

A request to the model is not just your message. As covered in Layer 2 of the Claude Code walkthrough, it is a system prompt, tool definitions, project context, the conversation so far, and finally the newest message.

Whatever nesting the JSON has, the API flattens it into one token sequence in a fixed order: tools first, then the system prompt, then messages in order. That order is what “prefix” refers to. Tool definitions are at position zero. The newest message is at the very end. (The order is a property of the API, not of how you write the JSON. On Anthropic’s API, tools renders before system even though most people write system first.)

So sort the contents of a request by how often they change, most stable first:

If the rendered order matches this stability order, caching mostly works on its own. If a per-request value sits at the front, nothing after it can ever be reused.

Step 3: Breakpoints

On Anthropic’s API you tell the system where a cacheable prefix ends by putting a cache_control marker on a content block:

const response = await client.messages.create({
  model: "claude-opus-5",
  max_tokens: 16000,
  system: [
    {
      type: "text",
      text: LARGE_STABLE_SYSTEM_PROMPT,
      cache_control: { type: "ephemeral" },
    },
  ],
  messages: [{ role: "user", content: question }],
});

A marker means “cache everything from the start of the request up to and including this block”. Since tools render before the system prompt, a marker on the last system block caches the tools too. You can place up to four markers per request. There is also a shortcut: a top-level cache_control on the request itself puts a marker on the last cacheable block for you, which is fine when you don’t need to control placement.

Only the last marker decides how much gets cached on this request. Earlier markers are fallback positions: if the next request changes something between marker two and marker three, the API can still match against marker two. And a marker on the newest message caches it for the next request, which is the basis of the multi-turn pattern in Step 6.

Prefixes shorter than the model’s threshold (roughly 1,024 tokens on most current models, as low as 512 on the newest and as high as 4,096 on a few) are silently not cached. No error, just cache_creation_input_tokens: 0. This is the first thing to check when a small prompt “isn’t caching”.

Why markers at all?

If a marker caches everything before it, why not always put one at the very end? For a plain conversation that is the right answer, and it is what the automatic top-level cache_control does. Writes cost more than reads, though, and storing a prefix costs the provider something, so the marker is how you say “this part is worth keeping”. Placing it yourself matters in three situations:

Put a marker where reuse ends, add one where a lookup would otherwise lose the trail, and give each stable region its own lifetime. Everything else can sit at the end.

Step 4: A request that misses, then one that hits

Two requests, 40 seconds apart, sharing a 12k-token system prompt with one marker on it. The first has nothing to reuse. The second does.

On every request the API checks whether the bytes up to your marker are already stored, reads them if so, and computes and stores them if not, while everything after the marker is computed either way.

With several markers, the API checks the last one first. If there is no entry for that exact prefix, it walks backward through the earlier markers and roughly the 20 content blocks before the marker, looking for the longest prefix that does exist. This lookback is what lets a marker move forward turn by turn (the next section shows that), and it has a limit: if a single turn adds more than about 20 blocks, say an agent firing many tool calls at once, the next request’s marker can fail to find the previous entry. The fix is an intermediate marker inside that window.

Step 5: The life of an entry

The default time-to-live of an entry is 5 minutes, and there is an opt-in 1-hour TTL. Every time an entry is read, its clock resets at no extra cost. If nothing reads it before the TTL runs out, it expires and the next request has to write it again.

Writes and reads are priced differently from plain input:

Step through a short burst of traffic and watch the entry get written, refreshed, and expire:

The 1-hour TTL is for traffic with gaps. If your requests arrive more often than every five minutes, they keep the entry alive on their own and the doubled write price buys you nothing.

Step 6: When it pays off

For the 5-minute TTL, one write plus one read costs 1.25 + 0.1 = 1.35x, against 2x for two uncached requests, so it pays off from the second request. For the 1-hour TTL, you need three requests (2 + 0.1 + 0.1 = 2.2x versus 3x). Beyond that, savings scale with how big the shared prefix is compared to the part that changes.

Set requests to 1 and you can see the one case where caching costs money: a prompt that is different from the beginning every time has no reusable prefix, so adding a marker only adds the write premium, and those prompts are better left unmarked.

The most important case in practice is the agent loop. Every turn appends to the conversation and changes nothing before it, which is the ideal shape for a prefix cache. The pattern is to put a marker on the last block of the newest turn on every request. Each request then reads everything from the previous turn’s entry and writes only the new tail. Here is the same nine-turn session from the token compounding diagram, with and without that moving marker:

The billed token count is identical in both modes. What changes is that most of those tokens are reads at a tenth of the price, and the prefill for a 50k-token context becomes a load instead of a recompute. Tools like Claude Code and Cursor are built around this: the system prompt and tool list stay frozen for the whole session, and anything that varies per turn goes at the end of the conversation, so the long prefix keeps hitting.

Step 7: What breaks it

Because the cache is a prefix match over the rendered sequence, the damage from a change depends on where in the sequence it lands. Tools render first, then system, then messages, and a change invalidates its own tier and everything after it.

Some of these have workarounds. Toggling tool_choice or thinking, or attaching an image, only touches the message tier, so it is fine to vary per request. Editing the top-level system prompt mid-conversation is expensive, but on newer models you can append a { role: "system", content: "..." } message inside messages instead, which lands after the cached history and leaves it intact. Adding or reordering tools, and switching models, are the two changes that rebuild everything, so keep the tool set fixed within a conversation and serialize it in a stable order.

A fork (a subagent, a summarizer, a compaction call) spins up a separate request. If it rebuilds system or tools with any difference from the parent, it misses the parent’s cache entirely. Copy the parent’s system, tools, and model verbatim, then append the fork-specific content at the end.

The silent invalidator

The most common failure is a small dynamic value in the wrong place. The markers are in place and nothing errors, and the response still never reports any reads.

Any of these inside the prefix does the same thing:

The fix is always one of three: move the dynamic piece after the last marker, make it deterministic, or delete it if it wasn’t doing anything.

Step 8: Checking that it works

Three fields in usage add up to the total prompt size and tell you what happened:

Log all three fields, not just input_tokens. On its own, input_tokens is only the uncached remainder, so an agent that ran for an hour can show input_tokens: 400 while the real prompt was 90k. And alert on cache_read_input_tokens being zero across repeated requests with what should be the same prefix. When that happens, diff the rendered request bytes between two calls, and whatever differs is what is invalidating the cache.

If a request is not creating entries either (cache_creation_input_tokens: 0 as well), the usual causes are a prefix under the minimum size, or a marker that isn’t reaching the API because the SDK call is building the request differently than you think.

The two ratios to watch

Per-request fields are for debugging. Over a day or a week, two ratios derived from their totals matter more, and Anthropic’s console shows both:

• Cache read ratio: read tokens divided by read plus uncached tokens (the console’s definition). Of the input that was not being written, this is the share that came from cache instead of being recomputed. Agent-heavy traffic in good shape sits around 85 to 90 percent.
• Write amortization: read tokens divided by written tokens, or how many times each cached token got read back. This says whether the write premium is paying for itself. The break-even is low: with the 5-minute TTL a written token only needs to be read about 0.3 times to come out ahead (1.25 + 0.1R against 1 + R), and about 1.1 times with the 1-hour TTL. Anything in the single digits or higher means writes are close to free in comparison.

The presets in the diagram are the common shapes. A high write count with almost no reads is the timestamp bug from earlier, at fleet scale. Zero writes and zero reads means the markers are not doing anything at all. And when the console breaks miss tokens down by reason (messages changed, system changed, tools changed, model changed), that is exactly the invalidation hierarchy from Step 7 measured on your own traffic: a spike in “system changed” or “tools changed” points at something upstream of the conversation being rebuilt per request.

Step 9: Parallel work and cold starts

A cache entry becomes readable only once the first request that writes it starts streaming its response. Requests that start before that point find nothing and each write their own copy.

For a burst of parallel requests over the same prefix, send one, wait for its first token, then send the rest. For interactive products where the first request of the day is user-visible, you can pre-warm at startup with a max_tokens: 0 request that carries the same system prompt and markers as real traffic. It runs prefill, writes the entry, and returns immediately with no output. Skip pre-warming when traffic is continuous, since real requests keep the entry alive on their own.

The rules in one place

• The cache is a prefix match over the rendered sequence: tools, then system, then messages. Any difference invalidates everything after it.
• Order content by stability. Frozen things first, per-request things last.
• Put a marker at the end of the stable region, not the end of the whole prompt. In a conversation, move a marker to the end of the newest turn on every request.
• Writes cost 1.25x (5m) or 2x (1h), reads cost 0.1x. It pays off from the second request.
• Keep tools and model fixed within a conversation. Forks must copy the parent’s prefix exactly.
• No timestamps, IDs, or conditional sections in the prefix. Make serialization deterministic.
• Per request, watch cache_read_input_tokens. If it stays at zero, something in the prefix is changing. Over time, watch the read ratio and the write amortization; anything above roughly 0.3 reads per written token (1.1 on the 1-hour TTL) is already paying off.
• For fan-out, let the first request start streaming before sending the rest.

The model’s work on a prompt can be saved from position zero up to the first thing that changed. Everything above is about arranging prompts so that the thing that changes comes last.