Prompt Caching

How It Works

When you send the same prompt prefix across multiple API calls, the upstream provider recognizes the duplicate, skips re-processing, and charges you a discounted rate for the cached portion. Typical savings:

Caching is enabled by default on all supported models — no configuration needed. The provider manages cache lifecycle automatically (caches typically persist for 5–30 minutes, varying by provider and load).

Cache-Friendly Prompt Design

The cache matches on prefix — tokens from the beginning of your messages array. Design your prompts to put everything static at the front:

# ✅ GOOD: Static content first = high cache hit rate
messages = [
    {"role": "system", "content": "You are a legal assistant. Reference case law when answering..."},
    {"role": "user", "content": "What are the elements of negligence?"},
]

# ❌ BAD: Dynamic prefix kills cache
messages = [
    {"role": "user", "content": "What are the elements of negligence?"},  # Cache miss
    {"role": "system", "content": "You are a legal assistant..."},  # Too late
]

Design Checklist

• System message first — Always put it as the first element in messages
• Static context before dynamic queries — Few-shot examples, retrieved RAG context, tool definitions go before the user's current question
• No timestamps / IDs in prefixes — Don't prepend unique per-request data before cacheable content
• Keep system prompts identical — The entire prefix must match byte-for-byte to hit the cache
• Longer prefixes = bigger savings — Caching a 10K-token system prompt saves far more than a 200-token one

Monitoring Cache Hits

The response usage object reveals whether your prompt hit the cache:

• Claude (Anthropic): Look for cache_read_input_tokens and cache_creation_input_tokens
• GPT (OpenAI): Cached tokens are reflected in lower prompt_tokens billing
• Gemini (Google): Context caching shows in the usage metadata

import requests

response = requests.post(
    "https://api.tokspan.com/v1/chat/completions",
    headers={"Authorization": "Bearer sk-your-key"},
    json={"model": "MODEL_NAME", "messages": [...]},
)

# Check for cache hits in the usage object
usage = response.json()["usage"]
if "cache_read_input_tokens" in usage:
    print(f"Cache hit! {usage['cache_read_input_tokens']} tokens served from cache")
    print(f"Cache creation: {usage.get('cache_creation_input_tokens', 0)} tokens written")
else:
    print("Cache miss — all prompt tokens billed at full price")

Supported Models