One timestamp, one round-robin, one floating tools list: 7 anti-patterns that kill the LLM prefix cache

In the process of preparing the article on caching economy, I gathered several anti-patterns that can break everything. At first, I thought it would be a short snippet at the end, but after reviewing the notes and doing a couple of researches, it became clear that this is a completely separate topic.

There are many such breakdowns with prefix_cache, but their mechanics are similar. In this article, I tried to reduce it to three reasons: requests stop matching the beginning, identical requests hit different machines, or the warmed-up cache doesn't survive until the next request.

Last time I wrote about money. Here, it's about why caching seems to be enabled, but there's almost no reuse: cached_tokens don't grow, latency increases, and the system pays the full price for prefill again.

TL;DR

• The most expensive failures are often caused not by user text, but by the wrapper around the API: tools, response_format, json_schema, serialization, mode switching.

• In a cluster with multiple replicas, a stable prompt by itself doesn't help. If you have round-robin without binding to the warmed-up cache, identical requests start warming up independently on different machines.

• In your infrastructure, it’s even more fun: even a good common part of the request can be lost due to eviction, block storage peculiarities, and too small a KV budget.

Quickly about the mechanics: three conditions without which the cache won’t work

For the cache to work at all, it’s not enough to have a “similar” request. Three conditions must be met simultaneously.

First, the requests must literally match the beginning. Not “almost the same,” but the same stream of tokens up to the point where the system can use the already calculated prefix.

Second, the request must go to a place where this beginning has already been warmed up. In a single-node setup, this is almost unnoticed. In a multi-replica setup, this is already a separate engineering problem.

Third, the KV-cache itself must survive until the next request. If it has already been evicted, an exact match of the beginning won’t help you.

Everything else is implementation details.

Synthetic case, which we will break everything on next

Let's take a corporate AI assistant. In each request, it sends a long system prompt, 14 tool definitions, and a short chat history tail. This is exactly the scenario where caching should work well: the general beginning part is long, there are many repetitions, and the cost of a miss is noticeable.

Before the release, everything seems fine:

Then the team releases three “harmless” changes:

1. It adds the current time and user.company to the system prompt.

2. tools starts gathering from a dynamic structure without a strict order fix.

3. After scaling to 4 replies with the model, it keeps the usual round-robin.

After the release, the model is the same, the tokens are almost the same, but the picture is already different:

Below is a synthetic but realistic case: the numbers here are illustrative. This is just a model of failure with an approximate order of magnitude.

In this case, all three conditions break at once: the beginnings of requests no longer match, identical requests go to different machines, and the warmed cache doesn't survive until the next call.

1. Volatile data at the start of the request

This is the most frequent killer of the hit rate. And the most trivial.

The problem looks like this:

system = f"""
Today: {datetime.now().isoformat()}
You are the corporate assistant for {user.company}.
{BASE_SYSTEM_PROMPT}
"""

It seems like normal personalization. In practice, you make each request unique from the very first tokens. One timestamp at the beginning—and the entire tail of the request is already different. The same happens with user.name, session_id, request_id, and any other fields that change more often than the cache survives.

The rule here is simple: everything that lives long and repeats between requests should be as close to the beginning as possible. Everything that changes from request to request should be as far from it as possible.

system = BASE_SYSTEM_PROMPT

user_ctx = {
    "company": user.company,
    "now": datetime.now().isoformat(),
}

messages = [
    {"role": "system", "content": system},
    {
        "role": "user",
        "content": f"[ctx={json.dumps(user_ctx, sort_keys=True)}] {query}",
    },
]

This does not make the entire request identical. But it keeps as common the part that is the most expensive in prefill.

2. Invisible template drift: when the same request is assembled differently in the system

An extra space in the template by itself is not a catastrophe. If you globally change the prompt once, the cache will just cool down once and then warm up again on the new version.

The problem starts elsewhere: when several almost identical variants of the same request exist in the system at the same time. One service adds one newline, another adds two. One path sends an image with detail="low", another with detail="high". One SDK serializes the message one way, another slightly differently.

To the eye, it’s “the same prompt.” For the cache — it’s no longer the same.

# web-chat
system = BASE_SYSTEM + "\\n"

# api-agent
system = BASE_SYSTEM + "\\n\\n"

or like this:
python
content = [
    {"type": "input_text", "text": user_query},
    {"type": "input_image", "image_url": img_url, "detail": "low"},
]

And in a neighboring code path, it’s already detail="high" or the same URL but with a different query string.

This is already the persistent fragmentation of the cache between several almost identical versions of the start of the request.

The most unpleasant part of this breakage is that people often look for it in the wrong place. They start debugging the cache itself, even though the problem is in the templating system, prompt rendering, SDK, or multimodal input serialization.

Therefore, it is necessary to normalize not only the user input but also the way the request is assembled: spaces, newlines, markdown templates, image parameters, and link serialization.

3. Tools, schema, and response_format are also part of the start of the request

Very often, it is not the prompt itself that breaks the cache, but the usual API wrapper around it.

tools = list(tool_registry.values())  # order depends on build/registration

response_format = {
    "type": "json_schema",
    "json_schema": {
        "name": "answer",
        "schema": {
            "type": "object",
            "properties": {
                "requestId": {"type": "string", "const": request_id},
                "answer": {"type": "string"},
            },
            "required": ["requestId", "answer"]
        }
    }
}

From an engineering standpoint, everything is valid. From the perspective of prefix caching, you made each request unique even before the user message.

The fix here is boring. And that’s why it works:

tools = canonicalize_tools(STABLE_TOOLS)  # fixed order

response_format = {
    "type": "json_schema",
    "json_schema": {
        "name": "answer",
        "schema": {
            "type": "object",
            "properties": {
                "answer": {"type": "string"},
            },
            "required": ["answer"]
        }
    }
}

And request_id, tenant context, trace id, and other live telemetry should be passed separately, not inside the cached schema.

This is an expensive failure because the schema and tools often weigh hundreds or thousands of tokens. One random requestId in the json_schema and the heavy block at the start of the request is no longer shared.

4. Rewriting history instead of appending (append-only)

The cache loves dialogue that grows at the tail. It does not handle request lists where you rewrite the beginning very well.

The problem typically arises when the team starts “optimizing” long context. Summarizing early moves. Aggressive compression of history. Naive trimming. Switching modes through the mutation system prompt.

messages = [
    {"role": "system", "content": DEBUG_SYSTEM_PROMPT},
    {"role": "assistant", "content": summary_of_previous_turns},
    {"role": "user", "content": new_query},
]

If you want to maintain reusability, it's better to grow at the tail:

messages = original_history + [
    {"role": "assistant", "content": summary_for_future_reference},
    {"role": "user", "content": new_query},
]

There’s also a more direct way to look at it: if you're really rebuilding the context from scratch, treat it as a new session and don't expect the old hit rate.

Summarization is not a bad idea in itself. Sometimes it's necessary. But it’s not a free speedup. It’s a trade-off: you reduce the context length at the cost of the cache no longer matching from the rewrite point.

5. The regular load balancer spreads the warm cache across replicas

Even a stable start to the request is only half the task. The second half is getting to the place where this request has already been warmed up.

The process doesn’t end with just assembling the prompt. You can stabilize the system prompt, canonicalize the tools, fix the schema – and still not get reuse if identical requests are distributed across different replicas.

router = RoundRobinRouter(replicas=4)
resp = router.send(request)

Practically, you need either a prompt_cache_key, or routing with warm cache consideration, or at least sticky routing at the gateway level:

resp = client.responses.create(
    model=MODEL,
    input=messages,
    tools=tools,
    prompt_cache_key=f"assistant:v3:tenant:{tenant_id}",
)

This has a cost: routing with cache binding improves reuse, but can create hot spots on popular prefixes. So you win in reuse, but you may lose in load balance uniformity if you don’t watch the balance.

This is not a theoretical criticism of the architecture. In the production stack vLLM, switching to prefix-aware routing raised the hit rate to 87.4% and gave about 70% savings in computations. So, proper routing here affects just as much as careful prompt assembly.

6. Parallel execution on the not yet warmed-up start of the request

This is a very counterintuitive bug.

The engineer thinks like this: “Now I’ll send 10 identical long requests, the first will warm up the cache, the others will win.” In practice, you can easily get the opposite result.

The problem isn’t in the stability of the prompt. The prompt can be identical. The order of execution is not.

tasks = [call_llm(shared_prefix, q) for q in batch]
results = await asyncio.gather(*tasks)

The working scenario is to first warm up the shared portion at the start of the request, and only then fan out similar calls:

# 1. Warm up the shared prefix
await call_llm(shared_prefix, warmup_query)

# 2. Only then fan out similar queries
tasks = [call_llm(shared_prefix, q) for q in batch]
results = await asyncio.gather(*tasks)

This anti-pattern is unpleasant because it is easy to confuse it with “somehow low hit rate with the provider,” while the problem is actually with the order and timing of your calls.

And here we smoothly transition to the next group of problems: even if the query reaches where it needs to, and even if the launch order is good, the cache still has to survive until the next request.

7. Cache does not survive to the next request

The last major group of failures is the lifetime of the cache.

At this point, everything can be done correctly at the request assembly and routing level, yet the hit rate may still be poor.

The first anti-pattern is rare traffic. If the "same" heavy query comes rarely, the usual in-memory cache will be almost cold each time.

The second one is that KV cache simply lacks memory. If more "warm" prefixes are living in traffic at the same time than can fit in memory, they start evicting each other. From the outside, this looks like a random hit rate: sometimes it’s there, sometimes it’s not. In reality, the system is repeatedly warming up the same parts of the query.

So, for managed APIs, this is a story about a short cache lifetime window. For your own infrastructure, it's also about the settings of the inference engine and the actual available memory.

What to monitor to notice the problem before the count

cached_tokens is a useful metric, but it shows the problem too late. When it noticeably drops, the latency and cost have usually already increased.

I would look at at least four things.

Firstly, what portion of the traffic can be cached at all. If a noticeable part of the queries is too short, unique, or does not have a common prefix, no cache optimization will give a significant effect.

Secondly, the time to the first token for groups of similar queries. If you have scenarios where queries should reuse the common prefix well, they should be monitored separately. Otherwise, on average, everything might look fine, but one important case will be broken.

Thirdly, behavior after changes: deployment, A/B test, changing the number of replicas, switching SDK or prompt template. A good dashboard should answer the question: after which exact change did the cache start performing worse.

Fourthly, how many times you rewarm the same pieces. In managed API this looks like jumps in hit rate and latency on common long requests. In your own infrastructure – also as a filled KV-cache and eviction of useful blocks.

Pre-deployment checklist

If I had to leave one block that people would actually take to production, I would leave this one.

Request start

• Only static and general content at the beginning.

• All personalization, time, request id, tenant-specific details – at the end or in metadata.

• One canonical rendering of the system prompt.

API wrapper

• tools are stable in content and order.

• JSON/schema are serialized deterministically.

• response_format does not contain dynamic fields like requestId.

• Mode switching does not mutate the base system prompt.

History

• The dialogue grows append-only.

• Summarization and compaction are a conscious trade-off, not a “free optimization.”

• Truncation in long dialogues does not break the most expensive common piece at the start of the request.

Routing

• For long common prefixes, there is stickiness: prompt_cache_key, affinity, prefix-aware routing.

• Parallel execution does not start before the first request has warmed up.

Cache lifetime

• KV budget corresponds to the working set, not the “maximum context from the model passport.”

• There is observability on eviction and rewarming, not just on the average cached_tokens.

To sum it up in one sentence: first stabilize the request start, then make sure identical requests go to the same machine, then ensure that the warmed cache even survives until the next request.

Conclusion

There are many anti-patterns, but almost all of them boil down to three failures: requests stop matching at the start, identical requests go to different machines, or the warmed cache disappears too quickly.

And this is the main practical conclusion. A bad prefix_cache_hit is a very specific engineering failure: one timestamp, one floating tools list, one round-robin, one too small KV budget.

If the topic resonates, I usually analyze such engineering forks in my channel - where the problem looks like "something grew in latency," but the root is actually in one detail of the architecture.