LLM Semantic Caching: Cut Repeat Model Calls

Applications built on large language models often receive the same question many times in slightly different words: "How do I reset my password?", "I forgot my password, what now?", "password reset steps". Sending each one to the model costs money and adds latency, even though the answers would be nearly identical. LLM semantic caching addresses this by storing previous answers and reusing them when a new request means the same thing as an earlier one, judged by comparing embeddings rather than exact text. Done carefully, it can reduce cost and response times; done carelessly, it serves wrong or stale answers. This guide explains how semantic caching works, when it helps, how to choose a similarity threshold and how to avoid the risks, with a worked example.

Exact caching versus semantic caching

An exact cache stores responses keyed on the precise prompt text. It is simple and safe, but misses whenever wording changes even slightly. A semantic cache converts each prompt into an embedding, a list of numbers representing its meaning, and looks for previously answered prompts whose embeddings are close enough. If it finds one above a similarity threshold, it returns the stored answer without calling the model.

Many systems use both: an exact cache first, because it is cheap and never wrong about whether two prompts match, then a semantic cache for near matches.

When LLM semantic caching helps

Semantic caching works best when:

• Many users ask similar questions with stable answers, such as support FAQs or internal policy questions
• Answers do not depend on who is asking or on rapidly changing data
• A slightly generic answer is acceptable
• Model calls are expensive or slow relative to an embedding lookup

It works poorly for personalised answers, questions about live data, such as an order's status, multi-turn conversations where context changes meaning, and creative tasks where varied output is the point. The savings depend entirely on how repetitive your traffic is, so measure before promising any figure.

How it works

1. Normalise the prompt, for example trimming whitespace and removing irrelevant boilerplate
2. Compute an embedding using an embedding model
3. Search stored embeddings for the nearest match, scoped to the right context
4. If similarity exceeds the threshold, return the cached answer
5. Otherwise call the LLM, return its answer and store the prompt embedding and answer

For storage, small systems can keep embeddings in memory, while larger ones use a vector database or an extension such as pgvector for PostgreSQL. Open-source libraries such as GPTCache package these steps for LLM applications.

Scoping the cache correctly

A semantic cache must never return an answer that was right for someone else's context but wrong for this one. Scope each cache entry by:

• Tenant or customer, for multi-tenant applications
• User permissions, if answers draw on documents with access controls
• Model and prompt template version, so a changed system prompt does not serve old answers
• Language and any other parameter that changes the answer

In retrieval-augmented systems, also invalidate entries when the underlying documents change.

Choosing a similarity threshold

The threshold is a trade-off. Set it too low and different questions get the same answer; set it too high and the cache rarely hits. Choose it with data rather than guesswork:

1. Collect a sample of real prompt pairs
2. Label which pairs should share an answer
3. Measure similarity scores for each pair using your embedding model
4. Pick the threshold that keeps wrong matches acceptably rare, then review regularly

Thresholds are specific to the embedding model, so re-tune whenever you change it.

Worked example: caching an internal IT helpdesk assistant

Suppose a company runs an internal assistant that answers IT policy questions from a fixed knowledge base. The team adds a semantic cache in front of the model. This is a simplified, untested illustration; embed stands for whichever embedding model the team uses:

import numpy as np

class SemanticCache:
    def __init__(self, embed, threshold):
        self.embed = embed
        self.threshold = threshold
        self.entries = {}  # scope -> (list of vectors, list of answers)

    def _vector(self, text):
        v = np.asarray(self.embed(text.strip().lower()), dtype=float)
        return v / np.linalg.norm(v)

    def lookup(self, scope, question):
        vectors, answers = self.entries.get(scope, ([], []))
        if not vectors:
            return None
        sims = np.vstack(vectors) @ self._vector(question)
        best = int(sims.argmax())
        return answers[best] if sims[best] >= self.threshold else None

    def store(self, scope, question, answer):
        vectors, answers = self.entries.setdefault(scope, ([], []))
        vectors.append(self._vector(question))
        answers.append(answer)

The scope combines the knowledge base version and prompt template version, so publishing a new policy document or changing the prompt starts a fresh cache. The team labels 300 pairs of past questions, measures similarity and chooses the threshold at which none of the sampled pairs with different correct answers would have matched. They log every cache hit with both questions so reviewers can spot bad matches, and give users a "this didn't answer my question" option that bypasses the cache. The sample size and review process are the team's choices; the cache's effect on cost depends on how repetitive the questions turn out to be, which the team measures after launch.

Other ways to reduce LLM costs

Semantic caching is one tool among several, and often not the first to reach for. Shorter prompts and tighter retrieval, sending only the most relevant passages rather than whole documents, reduce the cost of every call. Routing simple requests to a smaller, cheaper model and reserving larger models for hard ones can cut costs without affecting quality where it matters. Some model providers offer their own caching of repeated prompt prefixes, which helps when many requests share a long system prompt. Batch processing suits work that does not need an immediate answer. Measure where your spending actually goes, by feature and request type, before choosing; a semantic cache gives little benefit if most requests are unique, while tighter prompts help everywhere. Combining approaches is common: an exact cache, a scoped semantic cache for frequently asked questions, efficient prompts and sensible model routing.

Monitoring and maintenance

• Track hit rate, latency and model calls saved
• Sample cache hits for human review of match quality
• Expire entries after a set time where answers can change
• Invalidate on document, model or prompt changes
• Watch for feedback indicating wrong answers

Common mistakes

• Sharing cache entries across tenants or permission levels
• Caching personalised or live-data answers
• Picking a threshold without testing on real prompts
• Forgetting to invalidate when prompts, models or documents change
• Promising savings before measuring traffic repetition

For assistants that answer from company documents, our guide to private RAG architecture covers indexing and access control. For another production LLM use case, see our guide to AI invoice data extraction.

Frequently Asked Questions

What is LLM semantic caching?

Reusing a stored LLM answer when a new prompt is similar enough in meaning to an earlier one, judged by comparing embeddings, instead of calling the model again.

How is semantic caching different from exact caching?

Exact caching only matches identical prompt text. Semantic caching also matches differently worded prompts with the same meaning, at the risk of occasional wrong matches.

How much can semantic caching save?

It depends on how often users ask equivalent questions. Measure your traffic's repetition and hit rate rather than relying on published figures.

What are the risks of semantic caching?

Serving an answer to a question that only looks similar, leaking answers across users or tenants, and serving stale answers after documents, prompts or models change.

Make your LLM applications efficient

We build LLM applications with caching, retrieval and monitoring designed for accuracy as well as cost. See our AI automation solutions, or talk to us about your AI project.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.