Production Optimization

Minimizing Latency

Use Connection Pooling

Reusing HTTP connections eliminates the TLS handshake overhead on every request (~50-100ms saved per call). The OpenAI SDK pools connections automatically, but for production, tune the pool size:

import httpx
from openai import OpenAI

# Production-grade client with connection pooling
client = OpenAI(
    api_key="sk-your-key",
    base_url="https://api.onerouter.app/v1",
    http_client=httpx.Client(
        limits=httpx.Limits(
            max_keepalive_connections=20,
            max_connections=50,
        ),
        timeout=60.0,  # total timeout
    ),
)

Always Stream for Interactive UX

Set stream: true on every user-facing request. Streaming delivers the first token in ~100ms instead of waiting 5-30s for the full response. See Chat Completions — Streaming for implementation.

Edge Routing (Automatic)

OneRouter's DNS auto-resolves api.onerouter.app to the nearest edge location. No configuration needed. For self-hosted deployments, deploy in your application's region for sub-5ms network overhead.

Leverage Prompt Caching

Prompt caching can cut time-to-first-token by up to 80% on repeated prompts. Place static content (system instructions, context) at the beginning of your messages array. See the Prompt Caching guide for details.

Latency Checklist

Minimizing Cost

Smart Model Selection

Not every task needs GPT-4o or Claude Opus. Route simpler tasks to cheaper models:

Set Spending Caps

Configure per-key monthly budgets in the Dashboard. Keys auto-disable when they hit the cap — no surprise bills. Set lower caps on development keys and tighter limits on keys shared with clients. See Key Scoping.

Use Cost-Optimized Model Suffixes

Append -cheap to any model name to auto-route to the lowest-cost provider for that model. For non-critical batch jobs, this saves 10–30% with no code changes.

Cost Checklist

Maximizing Throughput

Async + Batching

For bulk processing, use async clients and concurrent requests. OneRouter's infrastructure scales horizontally — your throughput limit is typically your rate limit, not the server:

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(api_key="sk-your-key", base_url="https://api.onerouter.app/v1")

async def process_batch(prompts: list):
    tasks = [
        client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": p}],
        )
        for p in prompts
    ]
    return await asyncio.gather(*tasks)

Concurrency Guidelines

As a starting point:

• Pay-as-you-go: Up to 50 concurrent requests (500 RPM limit)
• Enterprise: Custom concurrency — contact us for your limit
• Self-hosted: Limited only by your infrastructure

Monitor x-ratelimit-remaining-requests in response headers to gauge your headroom. If you routinely hit 80%+ of your limit, request a raise.

Production Reliability

Retry with Exponential Backoff

Network blips and temporary provider issues happen. Always wrap API calls in retry logic:

import time
import random
from openai import OpenAI, RateLimitError, APIError

def chat_with_retry(client, model, messages, max_retries=3):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(model=model, messages=messages)
        except RateLimitError:
            if attempt == max_retries - 1: raise
            # Exponential backoff with jitter
            wait = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)
        except APIError as e:
            if e.status_code < 500 or attempt == max_retries - 1: raise
            wait = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)

Configure Auto-Failover

Set up a cross-provider failover chain (OpenAI → Anthropic → Google) in the Dashboard. If your primary provider goes down, traffic routes automatically with zero dropped requests. See Auto Failover.

API Key Strategy

• Dev key: Low budget ($10/mo), restricted to cheap models, no IP restriction
• Staging key: Moderate budget ($50/mo), production model set, IP-restricted
• Production key: Higher budget, all models, IP-restricted to production servers

Rotate keys every 90 days. Use separate keys per client if you're a reseller.

Quick Reference: Model Suffixes for Production