Reliability
Failure handling
Three mechanisms run together: choosing a target, moving on when it fails, and taking it out of rotation while it is unwell.
Choosing a target
Targets are scored on a recent window of latency and error rate. Weights are recalculated on a fixed interval and read from cache on the request path, so scoring does not add measurable work per request.
- Weighted selection across keys and endpoints
- Optional session affinity for stateful flows
- Per-target concurrency and timeout limits
Moving on
Each attempt is classified before a decision is made. Transient failures retry against the same target with backoff; failures bound to a key move to a different key instead of waiting.
- Ordered fallback chains with independent budgets
- Retry with exponential backoff and jitter
- Failure classification recorded per attempt
Taking a target out
A failure that describes the target rather than the request holds that target out of rotation. It is re-checked by a single probe once the wait expires, and returns through a ramp rather than at full weight.
- Hold scoped to a key, or to one model of a key
- Growing wait with a ceiling per failure class
- Manual release from the dashboard or the API
How a failure is classified
The class decides whether a target is retried, rotated or held. Request-shaped errors never hold a target.
| Class | Retry | Hold |
|---|---|---|
| Transient (network, 5xx) | Same target, with backoff | No |
| Rate limit | Rotate key | Yes, short |
| Credential rejected | Rotate key, no backoff | Yes |
| Quota exhausted | Rotate key, no backoff | Yes |
| Model not accessible | Rotate key, no backoff | Yes, that model |
| Request rejected by provider | No | No |
Next step
Talk to us about your architecture
Tell us how your traffic is shaped today. We will walk through where a forward path fits, what the failure policy should look like, and how the gateway would be deployed in your environment.
No pricing on this page. Every deployment is scoped individually.
- A review of your current provider and key layout
- A failure policy matched to your error classes
- A deployment shape for your network