These pages describe the intended behaviour of the gateway. The open-source release and the enterprise edition are being split out; details will be updated when the first release ships.
Circuit breaking
How a failing target is held out of rotation, re-checked, and returned to service.
A target that keeps failing costs a request every time it is chosen. Circuit breaking removes it from rotation until something suggests it can serve again.
Failure classification
Every failed attempt is classified before any decision is taken. The class is derived from the provider’s own error code, type or exception name where one is available, and from the status code only when it is not.
| Class | Retry | Hold | Meaning |
|---|---|---|---|
| Transient | Same target, with backoff | No | Network error, or 5xx that does not report a rate limit |
| Rate limit | Rotate key | Yes, short | The key or the model is throttled |
| Credential rejected | Rotate key, no backoff | Yes | The key itself was refused |
| Quota exhausted | Rotate key, no backoff | Yes | The account is out of credit or over a cap |
| Model not accessible | Rotate key, no backoff | Yes, that model | This key cannot reach the requested model |
| Model retired | Rotate key, no backoff | Yes, that model | The provider removed it |
| Region blocked | Rotate key, no backoff | Yes, that model | The provider refuses the request’s location |
| Request rejected by provider | No | No | The request itself was refused |
| Unrecognised | No | No | No signal to act on |
The rule that matters: a failure is only held against a target when it says something about the target. A malformed request never holds a healthy target out of rotation — otherwise one bad caller could take a provider offline for everyone.
Hold scope
A hold is scoped as narrowly as the failure allows.
- A rejected credential holds the whole key, for every model.
- A missing deployment holds one model, leaving the key serving its other models.
How long a target is held
The wait grows on a ladder, starting short and doubling after each failed check, up to a
ceiling that depends on the class. A wait the provider stated itself — a Retry-After
header, or a structured retry hint in the error body — is used as given, with a floor so a
sub-second hint does not cause a request every second.
Jitter is applied so that a fleet does not check the same target at the same moment.
Re-checking
When the wait expires the target is due. A single request is allowed through to it; every other request keeps skipping it until that probe answers.
| Probe result | Effect |
|---|---|
| Serves | Hold released, target returns through a ramp rather than at full weight |
| Same refusal | Held again, one rung further up the ladder |
| A different failure | Recorded on its own; the hold is settled |
Network error or 5xx |
Held again on the same rung, because the answer said nothing about the target |
Manual release
A hold can be released from the API or the dashboard. The target returns to rotation immediately and starts a fresh ramp.
Editing a credential also releases its holds, on the basis that the provider’s earlier answers applied to the previous value.