Cache the Plan, Not the Answer

I expected a boring result.

I was benchmarking semantic caches for LLM applications, the kind that embed an incoming query, find the nearest cached one, and serve the stored answer if the similarity clears a threshold. The plan was to measure how much they save. Instead, I ended up measuring how much they cost, and the numbers changed how I think about caching entirely.

The headline: a 5% change in the underlying data made 55% of the cache’s hits wrong. And the obvious fix, invalidation, didn’t fix it. Let me walk through what I found, in the order I found it.

The setup

An LLM answer cache stores request → response pairs. A semantic cache (GPTCache popularized the pattern; today it ships in LangChain’s semantic caches, Redis LangCache, and countless in-house builds) goes one step further: it matches new queries to cached ones by embedding similarity, so paraphrases hit too.

To measure correctness, not just hit rate, I built a benchmark where the ground truth is knowable: streams of recurring tasks with Zipf popularity (a few hot tasks, a long tail), where a controllable fraction of recurrences drift: the task recurs but its true answer has changed, because the data underneath it changed. Think “recompute the March rollup” after the March numbers moved.

Every query gets scored against ground truth: 616,200 of them across the study, on gpt-oss-120B and reproduced on Qwen2.5-32B. Scoring is deterministic: every task carries a generated ground-truth value, and answers are normalized before matching. Both caches under test ran the same embedder and the same similarity threshold, so any difference is mechanism, not tuning.

Finding 1: answer caches rot much faster than drift

Here’s the first result. The x-axis is drift rate, the share of recurring tasks whose true answer changed between recurrences. The y-axis is staleness: the share of cache hits that served a wrong answer.

Share of cache hits serving a wrong answer, vs drift rate. At 5% drift, 55% of hits are already wrong. The curve is steep because popularity amplifies drift: hot items are asked constantly AND drift like everything else. Once a hot item drifts, every later hit serves the dead answer. (Hover for exact values; click a legend entry to toggle a series.)

The counterintuitive part is the amplification. Drift probability is per item, but staleness accrues per request. Your most popular cached entries are exactly the ones whose corpses get served the most.

(For reproducibility: those streams are 600 requests over 40 distinct tasks at Zipf s = 1.1, so the exact 55% scales with recurrence depth; embedder and thresholds are in the repo README.)

Meanwhile the metrics everyone actually monitors stayed green: the failing cache held a ~93% hit rate throughout. Everyone knows caches go stale; that is why TTLs exist. What I have not seen measured is the rate (5% drift poisoning most of the hot-item hits) or the second failure mode below, and hit-rate dashboards surface neither.

Finding 2: invalidation traded one failure for another

The systems-engineering reflex says: add invalidation. Track what each cached answer depends on, drop the entry when the dependency changes. So I did, with a perfect signal: every change detected, zero stale entries served.

Accuracy at the heaviest drift level tested (0.4): 0.16. Identical to no invalidation at all.

This is where the study got interesting. The workload that exposes it is one I think of as method-in-history: a long-running agent where the user established a procedure earlier in the conversation and later re-asks (“recompute it for the new numbers”) without restating the method. Completely normal for agents. The invalidating cache correctly drops the stale entry… and then re-derives from a re-ask that doesn’t contain the method. It gets the answer wrong, caches that wrong answer as fresh, and serves it on every subsequent request.

If you build fully-hydrated, stateless RAG pipelines this can look alien, so here is the bridge: the user issued a shorthand re-ask, and invalidation dropped the only cache entry that contained the full derivation. Nothing in the incoming request carries the method anymore; the system just deleted its only copy.

A good mental model is this: invalidation solves detection, not re-derivation. If the knowledge needed to re-derive lived in history, dropping the stale entry just converts “serving outdated answers” into “manufacturing wrong ones.”

Naming the failure: re-poisoning

You can make this precise by splitting every wrong serve with one question: was this answer ever correct?

• Outdated: the answer was correct once; the world moved. Classic staleness.
• Re-poisoned: the answer was never correct; it was re-derived wrong and cached.

Same accuracy, opposite diseases. The naive cache fails outdated (6,036 wrong serves, 72% outdated). The invalidation-only cache fails re-poisoned (3,389 wrong serves, 99% re-poisoned), and it manufactured twice as many never-correct answers as the cache with no invalidation at all (3,369 vs 1,671).

Accuracy cannot distinguish these two failure modes: at every drift level the two caches score identically to each other, both landing at 0.16 at drift 0.4. The split can, and each mode needs a different fix. If you operate an LLM cache, this is one extra bit worth logging per serve.

One denominator note, because the counts look paradoxical next to the equal accuracy: “wrong serves” counts only answers delivered from cache. The naive cache hits 93% of the time, so its wrongness arrives almost entirely as cached serves (6,036). The invalidating cache’s hit rate falls to 56% under drift; a large share of its wrongness arrives as freshly re-derived wrong answers rather than cached ones, which is how it delivers fewer wrong serves (3,389) yet ends at the same accuracy.

The fix: replay the reasoning

The observation that unlocked it came out of a review pass over the code: the cache was already storing the derivation the model produced (its visible working, not any hidden chain-of-thought) alongside every answer, and never using it. The method the re-ask omits was sitting in the cache the whole time. For hosted models that expose no working at all, the same slot holds a structured plan instead; the engine treats it as an opaque procedure either way, scoped by the entry’s dependency fingerprints.

So instead of serving the frozen answer (wrong under drift) or re-deriving blind (re-poisoning), the engine does a third thing on drift: replay. Inject the stored reasoning and apply it to the new inputs. A short, exploration-free completion. I call the system YORO: You Only Reason Once.

Accuracy vs drift on the method-in-history workload. Replay holds 0.96 flat across the entire range while both alternatives collapse to 0.16; the stateless no-cache baseline sits at 0.07 because it never saw the method at all.

The full policy is graduated: serve when the entry is fresh (zero tokens) → replay when its dependencies changed (the method still applies; only the inputs moved) → reason fully when the request is novel. One gate decides the tier.

Reusing reasoning is not itself new: Buffer of Thoughts, Metacognitive Reuse, and Analogical Prompting all reuse thought templates to improve quality. What I have not seen elsewhere is the cache framing around it: dependency invalidation, the outdated/re-poisoned taxonomy, and honest input/output token accounting; reuse made safe and accounted for, rather than reuse as a prompting trick.

Cost: it’s a dial, not a point

Replay has a second property worth calling out: since the model already has the method, you can turn its reasoning effort down.

Output tokens (thousands per stream) vs drift. Full-effort replay: 96% accurate at ~21% of no-cache tokens. Low effort: 92% at ~10%. The failing semantic cache is cheapest of all at ~4%; being wrong is very cheap.

The chart shows output tokens because that is where reasoning effort shows up; the injected procedure costs input tokens, and the released curves account for both sides separately. Counting everything, replay totals ~37% of no-cache (123k vs 335k tokens per stream at drift 0.4) and low effort ~28%; replay’s input is actually lower than no-cache’s, because serves cost zero input.

When I checked why low-effort replay loses those four points, all 242 of its misses in the sampled run were arithmetic slips, zero truncations. Effort is genuinely the dial.

Tradeoffs and limits

Three honest ones, measured rather than asserted.

The invalidation signal is not an oracle. YORO’s zero staleness depends on being told when dependencies change (a fingerprint header, in the proxy implementation). So I measured what happens as that signal degrades: staleness tracks the share of missed signals almost linearly, converging to exactly the naive cache’s 0.866 at zero signal. Nothing falls off a cliff, but nothing is free either. TTLs, for comparison, are a clock-guess at this same signal: too long and you accrue the missed-signal staleness above; too short and you burn hit rate on entries that were still valid.

Staleness vs the share of missed invalidation signals. I pre-registered two predictions before this run: staleness ≈ 0.5 at 50% missed signals, and exact convergence to the naive cache at zero signal. They landed at 0.497, and 0.866 predicted, 0.866 observed.

The gate trades hit rate for correctness. On look-alike queries (same template, different entity), the naive cache force-fits 37% of them and keeps its 93% hit rate at 29% accuracy. YORO declines them: brittleness (the share of requests force-fit into wrong reuse) stays at 0.000, but hit rate falls from 0.89 to 0.33 as near-misses rise. That’s the honest bill.

Brittleness (wrong reuse on look-alike queries) vs near-miss rate. The naive cache force-fits its way to 0.371; the gate holds 0.000, and pays for it in hit rate, not correctness.

It is not a model quirk. The same divergence reproduces on Qwen2.5-32B (4-bit quantized, running on a single consumer GPU), with quantization biasing against replay (a 4-bit model slips more on arithmetic) and replay dominating anyway.

The Qwen2.5-32B replication: replay holds ~0.74 while serve-only and the semantic cache fall to 0.16. The lower ceiling tracks the smaller model’s own arithmetic ability; the structural gap is unchanged.

When this does not help. Three cases, plainly: if each request restates its full method, ordinary invalidation is enough. If you have no dependency signal at all, YORO degrades to a gated semantic cache. If a turn carries tools or sampling, the safe default skips caching entirely.

The regime matters. The replay result is measured where the method lives in prior interactions. If every request restates its full context, a plain invalidating cache performs equally well on correctness. And replay is validated on multi-step arithmetic procedures so far; extraction rules, rubrics, and tool plans are untested.

It runs on a laptop

None of this needs datacenter hardware. The whole study cost about $50 in rented GPU time (an H100 plus a consumer RTX 5090 for the Qwen replication; 4-bit, one card). And it runs locally, verified live against a 35B GGUF via llama.cpp on my Mac. Two receipts, attributed precisely: the released proxy demonstrates the serve tier today (repeated asks serve in ~12 ms versus ~3.3 s upstream); the full serve→replay→reason lifecycle ran in the library engine, where on drift it picked the replay tier and produced the correct new answer at 47% of the cold-reasoning tokens. The proxy wires the replay tier in next release; until then, replay lives in the library and the benchmark.

pip install "yoro-cache[embed]"

# any OpenAI-compatible endpoint works as the upstream
llama-server -hf deepreinforce-ai/Ornith-1.0-35B-GGUF --port 8000
YORO_UPSTREAM=http://127.0.0.1:8000/v1 yoro serve      # listens on :8400
export OPENAI_BASE_URL=http://127.0.0.1:8400/v1

Cache entries only serve while their dependency fingerprints match (X-YORO-Deps: rollup.csv:9f3ab2); the safe default refuses to cache tool-bearing or sampled turns, because a stale hit inside an agent can corrupt real work.

What I’d take away

If you run an LLM answer cache: your hit-rate dashboard cannot see the failure that matters, a 5% drift rate is enough to poison the majority of your hits, and invalidation without a re-derivation story can leave you worse off than no invalidation at all. Log the one extra bit (was this served answer ever correct?) and you’ll know which disease you have.

And if the method behind your answers lives in conversation history, the cheapest correct thing you can do is the one this whole study kept pointing at: cache the plan, not the answer.

Everything is open and in the repo today: the proxy, the library replay engine, the benchmark harness, and every result curve behind these figures: github.com/ChaitanyaPinapaka/yoro-cache · yorocache.com