Every AI stack already caches something. The question is whether it caches the right thing. Google now enables implicit caching by default on Gemini 2.5 and newer models, and Anthropic’s prompt caching will resume from a marked prefix in your prompt. Both save real money, and both only fire when the request prefix repeats. Neither one catches the most expensive pattern in production: the same question, asked twelve different ways.
Prefix caching is solved. Paraphrase caching is not.
Gemini’s implicit caching is on without configuration for 2.5-generation models and later. It needs a minimum prompt of 2,048 input tokens on 2.5 Flash and Pro (4,096 on the newer models), passes the savings through automatically, and reports hits in usage.total_cached_tokens. Anthropic’s equivalent works on marked prefixes via cache_control, either automatically or at explicit breakpoints.
That is genuinely useful. Long system prompts, tool schemas, retrieval context and multi-turn history all bill cheaper. But it is a string-prefix cache. Change one word ahead of the breakpoint and you pay full price, which is exactly what real users do. Redis makes the point with three prompts that mean the same thing and share almost nothing at the token level: “What are the features of Product A?”, “Can you list the main features of Product A?”, and “Tell me about Product A’s features.”
What a semantic cache actually does
A semantic cache, the design GPTCache popularised and the shape of nearly every offering since, inserts one step ahead of the model call. Embed the incoming query, look for a near neighbour among embedded past queries, and if the closest match clears a similarity threshold, return the stored answer without calling the model at all. Miss the threshold and the request proceeds normally, then the query embedding and the fresh answer get written back for next time.
Two maintenance facts before you adopt anything. GPTCache itself is MIT-licensed and ships a server image usable from any language, but its own repository now states the project no longer adds support for new model APIs, so treat it as a reference implementation to learn from rather than a component to build on. Managed options such as Redis LangCache trade that upkeep for a REST endpoint, and remain in preview.
Where the savings are, and how large
Semantic caching pays in proportion to how repetitive your traffic is, and support-style traffic is wildly repetitive: onboarding questions, policy lookups, “how do I…” requests, internal knowledge search. A human reads those as one intent. A token cache reads them as hundreds of distinct requests.
Redis publishes a deliberately simple model: monthly savings = monthly output-token cost x cache hit rate. In their worked example, a $200 monthly bill with 60% output tokens ($120) and a 50% hit rate yields $60 a month saved. Two honest caveats. First, serving from cache avoids output tokens; the input side is only partly offset because you now pay for an embedding plus storage on every request. Second, the hit rate is a property of your traffic, not of the tool. Measure your own repeat rate before believing any headline multiple.
The threshold is the whole product
One number decides whether this is a cost saving or an incident. Set the similarity threshold too loose and the cache serves a confident, plausible, wrong answer, which is the worst failure mode in production because nothing errors and nobody notices. Set it too tight and the cache almost never fires, and you have added latency plus a database for nothing.
It cannot be chosen by taste. You need a labelled set of queries with known duplicate and not-duplicate pairs, then you measure precision (how often a served hit was genuinely the same intent) and recall (how many real duplicates you caught). Research shows small implementation choices move those numbers materially: the MeanCache paper reports roughly 17% higher F-score and about 20% higher precision than GPTCache in an end-to-end deployment, plus an 83% reduction in storage from embedding compression. If you have not built the evaluation set yet, start with a golden set before you tune retrieval; the same set tunes your threshold.
Rules for a cache you can defend
- Never cache account-specific answers. A cached price is a wrong price, and a cached balance belongs to whoever asked first.
- Never cache time-sensitive facts. Stock levels, appointment slots, promotions and anything whose expiry actually matters.
- Namespace by tenant, then by user. One shared cache across client accounts is a cross-tenant data leak with extra steps.
- Keep personal data out entirely. If a stored answer contains PII, your cache is a second copy of your most sensitive table.
- Set a TTL and honour it. Eviction is a feature; a cache that only grows is a slow liability.
- Log
cached: true/falseon every response and expose a purge. When a source document changes, you must be able to invalidate its answers immediately.
You probably already own the store
A semantic cache is a vector index with a TTL, which means you very likely do not need another database for it. Postgres with pgvector already does approximate nearest-neighbour search with cosine distance (<=>) and HNSW indexes, next to the JOINs and transactions the rest of your app depends on.
Plan the storage before you switch it on. A 1,536-dimension float32 embedding costs 6 KB per vector, so 100,000 cached queries is roughly 586 MB of raw vectors before index overhead, and a million is about 5.7 GB. Half-precision halves both, which is the first lever to pull when the index outgrows memory, the same trade-off that shows up in cutting index size and re-embedding costs.
What to do this week
Instrument before you install. Log normalised query text for a week and count how often an incoming request is a paraphrase of one already answered. Pick a single high-volume, low-risk intent class such as an FAQ or policy lookup, and pilot the cache there alone. Ship it with a conservative threshold, a TTL, per-tenant namespacing and a cached flag in your logs, then tune against your evaluation set instead of in production.
Cost control in AI has moved from the model layer to the data layer, and the teams that win it treat caching as retrieval infrastructure with an audit trail rather than a switch. If you would rather have someone build and instrument that layer for you, our team does exactly this work: start with web development and we will scope the smallest version that pays for itself.