These pages describe the intended behaviour of the gateway. The open-source release and the enterprise edition are being split out; details will be updated when the first release ships.
Providers and credentials
Adding an upstream service and the keys it is reached with.
A provider is an upstream HTTP service. It is identified by a name, a base URL, and the routes it accepts. Nothing about a provider is compiled into the binary — adding one is a console action.
Adding a provider
| Field | Notes |
|---|---|
| Name | Your label for it. Appears in metrics, logs and routing trails. |
| Base URL | Scheme and host. Paths come from routes, not from here. |
| Routes | The path patterns this provider accepts |
| Default region | Used by cloud signature schemes when a request carries no region |
A provider with no routes is never selected. Routes are what make it reachable.
Adding a credential
| Field | Notes |
|---|---|
| Type | Header, bearer token, or a cloud signature scheme |
| Value | The secret. Written once; only a fingerprint is shown afterwards. |
| Weight | Relative share of traffic, when the provider has more than one credential |
| Model allowlist | Optional. Restricts this credential to a set of models. |
A cloud signature credential signs the whole request rather than adding a header to it, so it must be the only credential on its provider.
Key pools
Several credentials on one provider form a key pool. The pool is what failover rotates through and what load balancing distributes across, so the pool is where resilience comes from: a provider with one credential has nothing to rotate to.
A credential can be disabled without being deleted. Disabling is the right move when a key is suspected of being compromised but you are not ready to lose its history in the metrics.
Model allowlists
When a provider operates several models under different keys, an allowlist keeps a request from being sent to a key that cannot serve its model. The gateway checks the allowlist before sending, so a mismatch costs no upstream call.
What the console shows per provider
- Requests, tokens and cost, broken down by model
- Error rate by failure class
- Which credentials are in rotation, and which are held
- The current effective weights