Same payload, two backends: Vertex cached nothing, Studio cached every turn

My Gemini cache hit count was zero. Not low, zero. Every turn re-sent the same large static prefix the model had processed seconds earlier, and I paid full input price for it, every single turn. Implicit caching is supposed to make a stable prefix nearly free. Mine never hit.

What makes this one worth writing down is that the answer wasn't where I kept looking, and the thing that finally cracked it was an objection I almost waved off.

Ruling out the obvious

I run my own persona on Gemini 3 Flash via Vertex AI, with a set of tools. Google has an open issue for implicit caching on this family (#482476152): tool definitions get folded into the cache key, so they can fracture it. That was suspect one.

Before blaming tools, I killed the boring explanation, a prefix that isn't actually stable. Caching keys on the front of the request, so one drifting byte near the start misses every time.

import { createHash } from "node:crypto";
const fp = (s) => createHash("sha256").update(s).digest("hex").slice(0, 12);
console.log("prefix:", fp(staticHead)); // every turn

Same hash, every turn. Byte-identical prefix, still zero cached. Not drift.

Then I went down the list, one controlled test at a time:

Every theory I could name, a test refuted it. The cache worked in every repro and failed only when I actually used it.

The noise that fooled me three times

This took so long because my own measurements were dishonest, and the cache was noisy on top of it. I'd report the best cache hit across a few calls, which hides every miss. When I looked at the whole distribution, the same byte-identical request flickered:

Vertex, identical request:    0%   93%   0%   0%   93%

That flicker let me talk myself into a theory a day. Non-determinism. Then the streaming API. One run came back a clean, perfect zero and I started writing it up before a re-run cut it in half. A noisy system hands you a tidy story on demand. I had to learn to re-run the result I was proud of, every time, before believing it.

The objection that cracked it

Here's where I was about to ship the wrong conclusion. I'd written that real chat "never repeats, so it can't cache," because the conversation changes a little every turn: a new summary, fresh context, the growing history, the new message.

That explanation is wrong, and seeing why is the whole point. Prefix caching exists precisely so a stable head caches even when everything after it changes. I build the prompt the way the feature wants it: the unchanging instructions at the top, the parts that shift each turn at the bottom. Static first, dynamic last. If a changing suffix were supposed to break the cache, prefix caching wouldn't be a feature.

So "the suffix changes" can't be the cause. The real question is narrower: when the head is identical and only the suffix moves, does the backend still serve the cached head? That's the entire promise of prefix caching. So I tested it the one way that removes the noise: the same payload, sent to both backends in the same window, interleaved, round after round.

Same payload, two backends

Same churning payload, interleaved, 10 rounds:
            Vertex     Studio
  round 1:    0%         0%    (cold seed)
  round 2:    0%        68%
  round 3:    0%        68%
   ...        0%        67%
  round 10:   0%        67%
  ──────────────────────────────
  Vertex:  0% every round
  Studio:  caches from round 2 on, ~67%

Identical bytes. Identical changing suffix. The only difference is the backend. Studio served the stable head on every turn. Vertex served it zero times, every turn.

That's the answer, and it's clean:

Vertex's implicit cache rewards a whole request that recurs. It does not reliably do prefix caching, the part where it serves an unchanged head when the suffix moves. Studio does. Real chat reuses the head every turn but never repeats the whole request, so on Studio it caches and on Vertex it flatlines.

The prompt was built correctly the entire time. Studio proves it, on the exact traffic I actually send, every turn. The cache that never hit was a backend under-delivering on the prefix-caching contract, not a prompt assembled wrong.

I'll mark the edge of what I can claim. Google's implicit-cache eligibility is officially opaque, so I can show you the behavior precisely but not Google's internal reason for it. The behavior is enough to act on.

The fix is a cache you control

Implicit caching is best-effort by Google's own description. Explicit caching is the version with a guarantee: you store the static head yourself, give it a name and a lifetime, and reference it each turn while the changing tail rides along live. It pins the head as its own cache unit, so the backend can't lose it when the suffix moves.

// store the stable head (and tools) once
const cache = await ai.caches.create({
  model,
  config: { contents: [head], tools },
});
 
// every turn: reference the cache, send only what changed
await ai.models.generateContent({
  model,
  config: { cachedContent: cache.name },
  contents: [tail],
});

One thing stood in the way, and it was mine. Explicit caching had been hardcoded off, with a comment that it was incompatible with tools. That used to be true and quietly stopped being true. It works with tools now.

I switched my setup over. Now the static head caches on every turn, deterministically, and my per-message cost dropped by more than half.

What I took from it

Test in pairs, in the same window. The single biggest mistake here was trusting one synthetic run at a time on a backend whose noise wanders by the minute. Sending the same payload to both backends in one window removed the only variable I couldn't control, and the result stopped wandering instantly.

Re-run the result you're proud of. This cache fooled me into three clean theories, each with a tidy repro, before the paired test settled it.

And when a vendor feature underdelivers on its own contract, you don't have to understand why. You have to prove it, with a side-by-side, and route around it. The prompt was right. The fix was making the backend honor what it wouldn't honor on its own.