Claude Prompt Caching: Why 20 Block Window Misses

Claude prompt caching in agent loops can miss the 20-block window because the cache is invalidated whenever the conversation exceeds 20 blocks of content, forcing a full recompute of the prompt on the next request. This article explains why that happens, how the cache actually works, and how to structure your agent loop to avoid frequent cache misses.

How Claude Prompt Caching Works

Claude’s prompt caching lets you store a prefix of your prompt (the system prompt plus early conversation turns) for a short period. When you send a new request with the same prefix, Anthropic reuses the cached computation instead of reprocessing those tokens, which reduces latency and cost. The cache is broken into blocks of 1024 tokens (or 4096 for some models). The “20-block window� refers to the fact that the cache is only valid for the last 20 blocks of the prompt—roughly 20,480 tokens—before the oldest blocks are evicted.

Why the 20-Block Window Misses in Agent Loops

In an agent loop, each iteration appends new messages to the conversation. Once the total prompt length exceeds 20 blocks, the oldest cached block is dropped. If your agent then sends a request that includes that dropped block (because it’s still part of the conversation), the cache is invalidated for the entire prefix, and the model must recompute everything from scratch. This is a common cause of cache misses: the loop grows the prompt beyond the window, and the next request no longer matches the cached prefix.

How to Choose the Right Cache Breakpoint

To minimize misses, set your cache breakpoint at a point that stays within the 20-block limit for most of the loop. For example, if your system prompt is 2000 tokens, put the breakpoint right after it. As the conversation grows, the breakpoint will eventually fall outside the window, but you can periodically reset the breakpoint to a later point (e.g., after the last 20 blocks) to keep the prefix stable.

• Keep the system prompt and initial instructions under 20 blocks (≈20k tokens).
• Place the cache breakpoint after the system prompt, not after every turn.
• If the conversation exceeds 20 blocks, consider truncating or summarizing older turns.
• Use a sliding window: send only the last N turns plus the system prompt.

Common Problems and Misconceptions

A common misconception is that the cache is invalidated as soon as the prompt exceeds 20 blocks. In reality, the cache still works for the last 20 blocks; only the earlier blocks are evicted. The miss occurs when your request includes those evicted blocks, because the prefix no longer matches. Another issue is that some SDKs automatically add a cache breakpoint to every message, which can cause the prefix to shift and invalidate the cache more often than necessary.

Comparing Cache Strategies for Agent Loops

There are two main strategies: static breakpoints and dynamic breakpoints. Static breakpoints are simple but fail when the conversation grows. Dynamic breakpoints adjust the breakpoint as the conversation grows, but they require careful management to avoid shifting the prefix. For long-running agents, a hybrid approach—using a static breakpoint for the system prompt and a dynamic one for the recent history—often works best.

Cost and Latency Implications

Cache misses are costly: they require a full recompute of the prompt, which increases latency and cost. In a loop, a miss on every iteration can negate the benefits of caching. By keeping the prompt within the 20-block window, you can achieve cache hit rates above 90%, reducing latency by up to 85% and cost by up to 90% for the cached portion. However, the cache itself has a write cost (about 25% more than a normal input token), so you should only cache prefixes that are reused multiple times.

Alternatives to Prompt Caching in Agent Loops

If caching is not effective, consider other optimizations:

• Use a shorter system prompt to stay under the window.
• Summarize old conversation turns into a condensed context.
• Use a vector store to retrieve only relevant past messages.
• Use a separate model call to compress the conversation before continuing.

Frequently Asked Questions

Why does my cache miss even when the prompt is under 20 blocks? The cache is invalidated if any part of the prefix changes. Even a single token difference (like a timestamp or a random ID) will cause a miss. Ensure your prefix is truly static.

Can I cache more than 20 blocks? No, the cache window is fixed at 20 blocks for the models that support it. You can only cache the last 20 blocks of the prompt.

Does the cache persist across different conversations? No, the cache is tied to a specific prompt prefix. Each unique prefix has its own cache entry, and the cache is evicted after 5 minutes of inactivity.

How do I know if my cache is being hit? The API response includes a cache_read_input_tokens field. If it’s greater than zero, you got a cache hit.

What happens if I change the system prompt mid-loop? The prefix changes, so the cache is invalidated. You’ll need to rebuild the cache with the new system prompt.

Related

• All Stand Skins in Bizarre Lineage?
• Mazda CX-50 Hybrid vs RAV4: Tech & No Markup?
• Partially Threaded Wood Screws: Uses Explained

in t | Finish time | min | 50 | 0
in d | Distance | km | 10 | 0
out pace | Average pace | min/km | t / d | 2
out speed | Average speed | km/h | d / (t / 60) | 2