TOPIC AI
Prompt Caching optimizing LLM API latency and token reuse costs
IN 10 SECONDS
Saving the context history. When sending massive prompt instructions (like full project codebases or reference docs) to an LLM, the provider caches the initial part of the prompt. Subsequent questions reuse that cached state, slashing costs and response wait times.
GOTCHA Caching requires a byte-for-byte identical prefix. Adding a single extra space, newline, or variable parameter at the start of your prompt will break the match.
HOW THE CACHE SAVES TIME
01 First query submits a 20,000-word codebase. The AI provider processes it and caches the processed memory state.
02 Second query sends a short question using the exact same codebase prefix.
03 Cache match the provider finds the pre-processed codebase in memory, skipping recalculation.
04 Fast output the AI responds instantly and charges you a discounted rate for the cached text.
POKE IT YOURSELF
curl -X POST -d '{"model":"claude-3-5-sonnet","system":[{"type":"text","text":"...","cache_control":{"type":"ephemeral"}}]}' https://api.anthropic.com/v1/messages — call API with explicit prompt caching
Drill this topic →
~70 sec read