Why llama.cpp Reprocesses the Whole Prompt After a Tool Call and How to Keep the KV Cache Warm
When a service uses a tool and sends a follow-up request to llama-server, llama.cpp may process the conversation from the beginning again.…
When your code sends repeated requests to Claude through the Anthropic Messages API, prompt caching lets Claude reuse the processed portion of an earlier prompt. That shared part of the prompt, called a prefix, can contain system instructions, tool definitions, or reference material. A time to live (TTL) sets how long Claude keeps the prefix in the cache.
Choose the 1-hour TTL when the same prefix is likely to be reused after more than five minutes but within an hour. Keep the 5-minute default when requests arrive often enough to refresh the cache. The longer TTL costs more to write, so it pays off when it prevents a later request from having to write the prefix again.
Anthropic’s guidance favors the 5-minute default when requests reuse a prefix at intervals under five minutes. Each time a request reuses the cached prefix, the cache read refreshes the cache without an additional cache-write charge. Frequent reuse keeps the entry available at the lower write price. Choose the 1-hour TTL when a useful request is likely to arrive after that five-minute window but before the hour expires. Anthropic’s prompt-caching documentation names long-running agents and conversations with pauses between user turns as reasons to extend the TTL.
For example, an agent that waits for a slow tool or a sub-agent can resume after its five-minute cache entry expires. The resumed request then has to write the shared prefix again. A 1-hour entry can preserve that prefix through the wait, as Anthropic’s engineering guidance recommends considering for these workflows.
A user who pauses before replying creates a similar pattern. If the user is likely to return within an hour and the conversation reuses a large prefix, the longer TTL can avoid a new write. If the next reuse is unlikely or will happen after an hour, the longer window is unlikely to help.
The TTL clock starts when the request that writes or reads the cache begins, not when Claude finishes responding. A long response therefore uses part of the cache window. Anthropic gives the example of a response that streams for four minutes: the next request must start within about one minute of completion to use a five-minute cache entry.
For the Anthropic API, the documented cache-write prices are multipliers of the model’s base input-token price. A cache read generally costs 0.1 times that price, with model-specific exceptions.
| Cache operation | Price relative to base input-token price |
|---|---|
| 5-minute cache write | 1.25× |
| 1-hour cache write | 2× |
| Cache read | Generally 0.1× |
The 1-hour write costs 60% more than the 5-minute write for the same prefix: 2× rather than 1.25×. But if the five-minute entry would expire before the next request, the longer TTL can avoid a second write.
Suppose the uncached input cost of a shared prefix is P, and one follow-up request arrives after five minutes but within an hour. Under the standard 0.1× read price, the 1-hour path costs 2P for the initial write plus 0.1P for the read, or 2.1P. If the five-minute entry has expired, the alternative costs 1.25P for the first write plus 1.25P for a new write, or 2.5P. In this example, the longer TTL saves 0.4P across the two requests.
That comparison assumes the same prefix, a successful cache hit, and the standard read multiplier. Actual savings depend on the model’s rates and the number and timing of reuses. Use the current pricing and TTL guidance for the model you selected; do not treat the example as a universal cost forecast.
A longer TTL helps only when a later request matches the cached prefix. Keep shared instructions and context unchanged, and put per-request information after the reusable portion. If you add an explicit cache breakpoint, a marker for where the reusable prefix ends, place it at the end of that stable portion. Changes before the breakpoint, such as updated tool definitions or inserted request-specific text, can prevent a match. Anthropic’s cache guidance also explains that the matching prefix must use the same model and be byte-identical.
Check that the prefix meets the selected model’s minimum token requirement. Claude can process a marked prompt that falls below the minimum without caching it, and the request may not return an error. After requests run, inspect usage fields that report token counts, such as cache_creation_input_tokens and cache_read_input_tokens, to see whether Claude wrote tokens to or read tokens from the cache. Those measurements show whether the longer TTL is preserving reuse in your traffic.
The support and billing rules for Claude calls vary by API provider. For example, Amazon Bedrock’s documentation lists supported TTLs by model and API, and says an eligible request is not guaranteed to get a cache hit. Check the current rules for your provider and deployment rather than applying Anthropic API pricing or support assumptions to every platform.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com