Advanced Features
Load Balancing
TokSpan automatically distributes your API traffic across multiple models and providers to maximize throughput, minimize latency, and avoid single-provider bottlenecks. No configuration required on your side.
Load Balancing Strategies
TokSpan automatically picks the optimal strategy for every request across its upstream providers:
| Strategy | How It Works | Automatically Applied When |
|---|---|---|
| Weighted Random | Requests are spread proportionally across healthy providers (e.g., 70% to one, 30% to another) | Spreading load across multiple healthy providers |
| Priority (Failover) | Traffic uses the healthiest provider first, failing over down the chain automatically | A preferred provider is healthy; others serve as automatic backup |
| Rate-Limit Based | Overflow is distributed to other providers when one hits its rate limit | A provider hits its rate limit during high-throughput workloads |
Zero configuration required. Load balancing is applied automatically to every request — you just keep calling the same model name.