The most direct way for AI startups to lower serving costs is to stop paying twice for the same tokens. Prompt caching, the reuse of already-computed model work for a repeated prompt prefix, lets later requests skip recomputation and pay a reduced cached rate. Semantic caching goes further by returning a stored answer for a paraphrased question. Together these methods help startups with repeated system prompts, documents, or agent histories.
Table of Contents
- What does automatic prefix caching save?
- When should you mark cache points yourself?
- What changes when you self-host?
- How do you catch similar questions?
What does automatic prefix caching save?
OpenAI prompt caching is automatic for supported models on prefixes of 1,024 or more tokens. It requires no code change. According to OpenAI docs, reused cached-input tokens receive up to 95% discount while time to first token falls OpenAI prompt caching guide. Google reports a similar default-on benefit for Gemini.
Its implicit caching automatically discounts repeated prefixes, advertised at 75% savings on Gemini 2.5 and documented at 90% off cached tokens on Vertex AI. The practical effect is lower cost for stable beginnings. A customer-support bot with the same policy section benefits on every call. A coding assistant with the same repository context does the same.
When should you mark cache points yourself?
Anthropic Claude uses cache_control with type ephemeral. Teams can let growing chat history cache automatically or set explicit breakpoints on stable blocks. Pricing rewards reuse. According to Anthropic, cache reads cost 0.1 times base input price, while cache writes cost 1.25 times for five-minute retention and 2 times for one-hour retention Anthropic prompt caching update.
Anthropic also reports savings up to 90% in cost and up to 85% in latency for long prompts. Use explicit breakpoints for large stable material. Cache the system prompt, product docs, and tool schemas. Keep changing user text and short-lived agent output outside cached blocks.
What changes when you self-host?
Self-hosted vLLM offers Automatic Prefix Caching with enable_prefix_caching set to True. According to the vLLM project docs, it hashes key-value blocks and maps identical-prefix blocks to shared physical memory vLLM Automatic Prefix Caching docs. That design skips shared-prefix computation across requests. Two users with the same system prompt share the first blocks.
Ten agent steps with the same history prefix share that history. The win grows with shared structure. Keep prefixes identical across users. Avoid user-specific IDs, timestamps, or random ordering at the start.
How do you catch similar questions?
Exact-prefix caches miss when the wording differs. Semantic caching with embeddings, such as open-source GPTCache, returns stored answers for paraphrased queries. According to the GPT Semantic Cache study, this approach cut up to 68.8% of LLM API calls at 61.6 to 68.8% hit rates with over 97% positive-hit accuracy GPT Semantic Cache study.
It fits high-volume near-duplicates. Repeated onboarding questions, status explanations, and product FAQs are strong candidates. Low-risk informational answers are safer than actions, refunds, or access changes. Set boundaries before launch:.
- Place stable instructions, schemas, and documents first so exact caches hit.
- Tune the similarity threshold on real startup queries to limit wrong answers.
- Isolate caches by tenant to avoid leaking one customer answer to another.
- Measure hit rate, accuracy, and net savings after cache-write costs.