Semantic Caching for LLM Apps: Cut Cost Without Serving Stale Answers

Exact caching asks whether two requests are identical. Semantic caching asks whether they mean the same thing.

That distinction can remove a shocking amount of model traffic. It can also return a confident answer to the wrong question, leak one tenant’s context into another, or keep yesterday’s policy alive after the source changed. Semantic caching is a relevance system wearing a performance hat. Treat it accordingly.

Where Semantic Caching Actually Helps

The best candidates are high-volume, low-volatility requests with stable answers: documentation explanations, product FAQs, classification, normalization, and repeated support questions.

Poor candidates include personalized advice, rapidly changing data, authorization-sensitive results, financial quotes, order status, and workflows whose answer depends on hidden state. If the correct response can change without the visible query changing, a query-only cache key is unsafe.

Measure duplication before building. Embed a privacy-safe sample of requests, cluster them, and estimate how often similar intents repeat. A cache that saves two percent of traffic may not justify its operational complexity.

Build a Complete Cache Identity

Do not embed only the user’s latest sentence. The response may depend on:

• system and developer prompt versions
• tenant and user permissions
• locale
• conversation state
• retrieved corpus version
• model and tool versions
• safety policy
• output format
• feature flags

Separate hard-key fields from semantic fields. Tenant, authorization scope, locale, and prompt version usually require exact equality. The user request and selected context can be compared semantically inside that safe partition.

Never share personalized entries across users just because the questions look similar.

Choose a Similarity Threshold With Data

There is no universal cosine-similarity threshold. The right value depends on the embedding model, domain, query length, and cost of a false hit.

Create labeled pairs:

• equivalent and safe to reuse
• related but not reusable
• clearly different

Plot precision and recall across thresholds. Optimize for precision because a false cache hit is often worse than a miss. Use a higher threshold for consequential answers and consider a lightweight reranker or rule check before reuse.

Short queries are especially dangerous. “Pricing,” “limits,” or “reset” can match many intents. Require additional context or bypass the cache when the request lacks enough semantic detail.

Store More Than an Answer

A useful cache entry includes the embedding, normalized request, response, creation time, expiration, tenant scope, prompt version, model route, source document IDs, source version, safety classification, and quality score.

Store provenance so you can invalidate entries when a document changes. If an answer cited policy version 12, publishing version 13 should retire or revalidate dependent entries immediately instead of waiting for a generic time-to-live.

Do not cache unvalidated failures. A fluent but unsupported answer becomes more harmful when served at scale.

Design Freshness Explicitly

Time-to-live is only one control. Combine:

• short TTLs for volatile domains
• event-based invalidation on content updates
• versioned namespaces for prompts and models
• source-aware invalidation
• manual purge controls
• background refresh for popular entries

Use stale-while-revalidate only where a slightly old answer is acceptable. Label the policy by use case. Product documentation may tolerate minutes; account permissions may tolerate nothing.

Keep Safety and Privacy Ahead of Hit Rate

Apply authorization before cache lookup. Partition indexes by tenant or access boundary. Encrypt sensitive entries, minimize stored content, and apply the same retention policy used for the underlying data.

Prompt injection can poison a cache if untrusted content influences a reusable answer. Validate source trust, record provenance, and avoid promoting a response into a shared cache until it passes grounding and safety checks.

Also consider deletion rights. If a response contains personal data that must be removed, you need to find every derived cache entry. Provenance is not optional.

Use a Two-Stage Lookup

A practical flow looks like this:

1. Build the hard-key partition from tenant, permissions, locale, prompt version, and output schema.
2. Normalize the request without erasing important numbers or entities.
3. Generate an embedding.
4. Search the safe partition for nearest candidates.
5. Apply a calibrated similarity threshold.
6. Verify freshness, source versions, and policy compatibility.
7. Optionally rerank the top candidate.
8. Return the cached response or call the model.
9. Validate new output before storing it.

Track why each lookup missed. A prompt-version miss suggests different work from a low-similarity miss.

Evaluate the System End to End

Monitor hit rate, accepted-hit precision, latency saved, model cost saved, stale-response rate, wrong-answer reports, and cache contribution to task completion.

Run shadow mode before serving cached answers. Perform the lookup, but still call the model. Compare the candidate with the fresh response using deterministic checks and human review on sampled pairs. This reveals false hits without exposing users.

Then canary a small traffic percentage. Keep an instant bypass switch and log cache decisions with entry IDs, similarity scores, versions, and invalidation state.

Avoid the Famous Traps

Do not lower the threshold to make the dashboard look good. Do not cache across tenants. Do not treat embeddings as authorization. Do not ignore numbers: “budget of $500” and “budget of $5,000” can be semantically close and operationally different. Do not let model or prompt changes silently inherit old answers.

The real goal is not maximum hit rate. It is maximum safe reuse.

Built carefully, semantic caching makes an LLM product feel instant and dramatically cheaper. Built lazily, it industrializes stale context. The difference is not the vector database. It is the discipline around identity, thresholds, provenance, and invalidation.