These pages describe the intended behaviour of the gateway. The open-source release and the enterprise edition are being split out; details will be updated when the first release ships.
Load balancing
Weighted selection across keys and endpoints, scored from observed latency and error rate.
When a provider has more than one credential, or a route has more than one target, the gateway chooses one per request.
Static weights
Each credential carries a weight, set in the console. Selection is weighted-random over the pool, so the long-run share of traffic matches the weights you set.
provider openai
├─ primary weight 3.0
└─ secondary weight 1.0The example above sends roughly three requests to primary for every one to secondary.
Health-aware scoring
Weights are not fixed. A recalculated score adjusts each target’s effective weight using three signals, in priority order:
| Signal | Role |
|---|---|
| Error rate | Primary. Recent failures matter more than old ones. |
| Latency | Secondary. A target is compared both to its peers and to its own recent baseline. |
| Utilisation | Tuning. Discourages concentrating traffic on one high performer. |
Scoring runs on a fixed interval and its output is read from cache on the request path, so selection adds no measurable work per request. The console shows the effective weight alongside the configured one, which is how you tell a target that is configured heavily from one that is merely performing well.
Why not always pick the best
Selection is probabilistic rather than greedy. A target with a low weight keeps a small share of traffic, which is how a recovered target is discovered instead of being skipped forever. A weight floor guarantees that no healthy target is starved.
Sessions
A session pins a sequence of requests to one target. Health wins over affinity: if the session’s target is held out of rotation, the request goes elsewhere rather than to a target known to be failing.
What load balancing does not do
It does not inspect request content to make a decision. Selection is based on the target’s recent behaviour and the configured weights, never on the prompt.