Failover and routing
How a failing provider falls over to the next route, and the provider object that steers the choice.
A model can have more than one route. When the first provider fails with a server error, a rate limit, or a problem with our account there, the request moves to the next route and you get the answer rather than the error. Whichever route served is the one billed, at its own price, and the response reports provider and fallbacks_tried.
A genuine mistake in your request is not retried. A malformed body or a refusal is returned immediately, because sending it to three more providers only wastes your time.
You can steer this per request with a provider object carrying only or ignore, or set a default in your dashboard. If your preferences leave no usable route, you get 503 rather than a silent substitution.
| provider field | What it does |
|---|---|
| only | Provider slugs. Only these routes are used. |
| ignore | Provider slugs. These routes are skipped. |
| order | Provider slugs, tried first and in this order; the rest follow. |
| sort | "price" (the default, cheapest first), "throughput" (most output tokens a second) or "latency" (lowest median latency), from the last 24 hours of served traffic. The suffixes :floor and :nitro on a model id mean price and throughput. |
| max_price | { "prompt", "completion" }, US dollars per million tokens. Dearer routes are skipped; if none is left the answer is 404 no_endpoints_within_max_price. |
| require_parameters | true keeps only routes known to honour the request's tools and response_format; if none is left, 404 no_endpoints_support_parameters. |
| allow_fallbacks | false tries the first route only. |
Defaults saved in the dashboard's Routing pane apply to every request, and a provider object in the request wins field by field. A key in a workspace with routing preferences of its own uses the workspace's instead of the account's.
{
"model": "meta-llama/llama-3.3-70b-instruct",
"provider": { "sort": "throughput", "ignore": ["deepinfra"], "max_price": { "prompt": 1, "completion": 1 } },
"messages": [{ "role": "user", "content": "Hi" }]
}