Production Optimization

Minimizing Latency

Use Connection Pooling

Reusing HTTP connections eliminates the TLS handshake overhead on every request (~50-100ms saved per call). The OpenAI SDK pools connections automatically, but for production, tune the pool size:

import httpx
from openai import OpenAI

# Production-grade client with connection pooling
client = OpenAI(
    api_key="sk-your-key",
    base_url="https://api.tokspan.com/v1",
    http_client=httpx.Client(
        limits=httpx.Limits(
            max_keepalive_connections=20,
            max_connections=50,
        ),
        timeout=60.0,  # total timeout
    ),
)

Always Stream for Interactive UX

Set stream: true on every user-facing request. Streaming delivers the first token in ~100ms instead of waiting 5-30s for the full response. See Chat Completions — Streaming for implementation.

Edge Routing

Requests to api.tokspan.com are served via TokSpan's edge network, which routes traffic to the backend region that hosts your model. No configuration needed on your side.

Leverage Prompt Caching

Prompt caching can cut time-to-first-token by up to 80% on repeated prompts. Place static content (system instructions, context) at the beginning of your messages array. See the Prompt Caching guide for details.

Latency Checklist

Optimization	Latency Impact	Effort
Connection pooling	−50–100ms per request	Low
Enable streaming	Perceived: −5–30s	Low
Prompt caching	−80% on cache hits	Medium

Minimizing Cost

Smart Model Selection

Not every task needs GPT or Claude Opus. Route simpler tasks to cheaper models:

Task Type	Recommended Model	Cost vs. GPT
Classification, extraction, tagging	GPT mini, Claude Haiku, Gemini Flash	10–50× cheaper
Drafting, summarization, translation	DeepSeek, Llama, Mistral	3–10× cheaper
Complex reasoning, code generation	GPT, Claude Opus	Baseline
Batch / background processing	DeepSeek or other low-cost models	5–15× cheaper

Set Spending Caps

Configure per-key spending caps in the Dashboard. Keys auto-disable once their quota is exhausted — no surprise bills. Set lower caps on development keys and tighter limits on keys shared with clients. See Key Scoping.

Cost Checklist

Optimization	Cost Impact	Effort
Route simple tasks to mini models	−70–95% on those tasks	Medium
Enable prompt caching	−50–90% on cache hits	Low
Set per-key spending caps	Hard cap on max spend	Low
Monitor usage dashboard weekly	Catch anomalies early	Low

Maximizing Throughput

Async + Batching

For bulk processing, use async clients and concurrent requests. TokSpan's infrastructure scales horizontally — your throughput limit is typically your rate limit, not the server:

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(api_key="sk-your-key", base_url="https://api.tokspan.com/v1")

async def process_batch(prompts: list):
    tasks = [
        client.chat.completions.create(
            model="MODEL_NAME",
            messages=[{"role": "user", "content": p}],
        )
        for p in prompts
    ]
    return await asyncio.gather(*tasks)

Concurrency Guidelines

As a starting point:

Pay-as-you-go: Start with modest concurrency (5–10 parallel requests) and scale up based on observed latency and responses
Enterprise: Custom concurrency — contact us for your limit

If you start receiving 429 responses, back off before retrying (exponential backoff with jitter) and reduce concurrency. Rate limits vary by plan and model — see Rate Limits for details.

Production Reliability

Retry with Exponential Backoff

Network blips and temporary provider issues happen. Always wrap API calls in retry logic:

import time
import random
from openai import OpenAI, RateLimitError, APIError

def chat_with_retry(client, model, messages, max_retries=3):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(model=model, messages=messages)
        except RateLimitError:
            if attempt == max_retries - 1: raise
            # Exponential backoff with jitter
            wait = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)
        except APIError as e:
            if e.status_code < 500 or attempt == max_retries - 1: raise
            wait = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)

Automatic Same-Model Failover

If the provider serving your model experiences downtime, TokSpan automatically reroutes requests to another provider serving the same model — with zero dropped requests and no manual intervention. Failover stays within the same model, so output behavior remains consistent. See Auto Failover.

API Key Strategy

Dev key: Low budget (e.g. $10), restricted to cheap models, no IP restriction
Staging key: Moderate budget (e.g. $50), production model set, IP-restricted
Production key: Higher budget, all models, IP-restricted to production servers

Rotate keys every 90 days. Use separate keys per client if you manage multiple projects.

Quick Reference: Production Checklist

Before going live, work through this checklist:

Use a production-grade client — connection pooling and explicit timeouts (see above)
Enable streaming on every user-facing request for a responsive UX
Implement retry with exponential backoff and jitter for <code>429</code> and <code>5xx</code> errors
Set per-key budgets and IP whitelists so a leak can't cause a large bill
Keep static prompt content first to maximize prompt-cache hit rates and cut costs