Why llama.cpp Reprocesses the Whole Prompt After a Tool Call and How to Keep the KV Cache Warm

When a service uses a tool and sends a follow-up request to llama-server, llama.cpp may process the conversation from the beginning again. Tool calling does not require this. The server can reuse an unchanged beginning of the prompt and process only the new tokens, the text units the model reads. To do that, it must still have valid access to the earlier model state.

That saved state is the KV cache: the attention keys and values a model computes while reading tokens. Keeping it warm means retaining and reusing that state across requests. To find why the server did not reuse cached state, compare the prompt it sends to the model across turns and check whether the server still has the right slot state. For hybrid or recurrent models, also check whether the build in use restores checkpoints correctly.

A tool call adds a turn to the prompt

In llama.cpp’s function-calling flow, the chat template, which turns conversation messages into model input, includes the assistant’s tool call in the conversation. The function-calling documentation describes how the client runs the tool, adds its result to the messages, and sends another request. The server then renders that conversation into tokens and checks whether its beginning matches state it already has.

If the tokens at the start of the prompt match those from most of the prior conversation, the server can reuse the corresponding cached state and process only the added end. If the rendered prompt changes near the beginning, the shared prefix ends there, so the server has less state it can reuse. The server’s cache_prompt behavior is designed to reuse matching prompt prefixes and is enabled by default, though request or server configuration can override it.

The server compares tokens rendered from the messages, rather than the client’s JSON or the messages’ meaning. Changes to system instructions or tool declarations can alter the token prefix. Changes to template output or serialization of earlier messages can alter it too. Check the rendered prompt or token IDs across requests to find the earliest difference. A tool result appended at the end does not, by itself, invalidate an unchanged prefix.

Why the server may lose reusable state

The server can reuse state only if it still has access to the slot holding that state. A slot is a server-managed place for a request’s prompt and model state. Requests assigned to different slots, or a slot whose state has been discarded, may lose an opportunity to reuse state held by a live slot.

The server README documents slot assignment and two related cache controls: --cache-ram, which sets the RAM available for saved prompt state, and --cache-idle-slots, which saves idle slots into that cache. These controls help preserve state across slot reuse; they do not repair a prompt whose rendered prefix has changed. First confirm that prompt caching is enabled and that requests follow the expected slot path. Change RAM cache settings only if the workload actually needs state restored after a slot becomes idle.

A user report alleges that RAM-backed cache restoration returned content from an unrelated conversation under concurrent load. That report is a reason to test correctness under the deployment’s own concurrency pattern, not evidence that every deployment has the problem. The report is tracked in issue #27148.

The server also has a separate --cache-reuse setting that attempts reuse through KV shifting. The README says it requires prompt caching and defaults to 0, meaning disabled. Treat it as a specialized option to test for a specific workload, not as a fix for changed prompt tokens or invalid model checkpoints.

Hybrid and recurrent models need valid checkpoints

For some hybrid or recurrent models, matching prompt text is not enough. These models maintain state beyond the ordinary attention cache, and the server must restore that state at a position where the saved checkpoint is valid. A checkpoint is a snapshot of model state at a particular position in the prompt.

llama.cpp’s server code handles checkpoint restoration, and issue #24055 reports a failure mode in which validity checks caused long tool-calling sessions on hybrid or recurrent models to fall back to full prompt processing. Check the server logs for messages saying it is forcing full prompt reprocessing because cache data is missing. If those appear while the rendered prefix remains stable, changing the chat template is unlikely to address the root cause.

A proposed fix in PR #25592 preserves the requirement that recurrent state be restored at the exact position where it was saved. As of November 10, 2026, the pull request is still marked open, so do not assume an upstream build includes it. Check the status of the build you run and test checkpoint behavior with the model and session length that reproduce the issue.

In older builds, a separate checkpoint-spacing issue caused more limited reprocessing, around one ubatch, a small batch of tokens processed together. The issue discussion distinguishes that bounded behavior from the full-context hybrid or recurrent problem and reports a fix in PR #25472. Do not treat every instance of extra prompt processing as the same checkpoint defect.

Keep the prefix stable and measure what the server processes

First check the request’s cache_prompt setting. Then compare the prompt sent to the model, or its token IDs, between consecutive tool turns. Keep system instructions, tool declarations, and earlier message formatting stable where possible; place changing content later in the prompt when the conversation design allows it. These practices preserve the common prefix that prefix caching needs, but they cannot guarantee reuse if the server has discarded the state or the model cannot restore it safely.

Next, check whether requests return to the intended slot and whether idle-slot caching fits the workload. If full reprocessing continues despite a stable prefix and available slot state, inspect server logs for checkpoint messages, especially with hybrid or recurrent models. llama.cpp tracks how many prompt tokens it processes and how many it reuses from cache. Those counts show whether it processed the whole prompt or only the new ending. The server’s token-count definitions provide context for interpreting those measurements.

A report attached to PR #25592 gives one indication of the potential impact, but not a general benchmark. On a patched build, the reporter measured a roughly 38,500-token prompt taking 40.8 seconds to process in full and 0.1 to 0.3 seconds after checkpoint restoration, an estimated 150 to 400 times faster. The report describes a particular model and setup; it does not establish results for other models, backends, or upstream builds.

Finally, cached prompt reuse has a documented trade-off: the server README warns that it can produce nondeterministic results on some backends because prompt processing and token generation can yield different logits. If deterministic output matters, test the exact backend and configuration rather than assuming caching leaves results unchanged.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com