Provider Fallback

Provider Fallback lets you attach a per-request chain of alternate (provider, model) targets. If your primary call fails with an availability-class error, LLM API transparently retries the request against the next target in the chain — no client-side retry loop, no duplicate billing on success.

INFO

Fallback is off by default. A request that does not carry a policy header behaves exactly as if this feature did not exist: one upstream call, primary error surfaced as-is.

At a Glance


What it does	Retries a failed upstream LLM/audio/video call on a different `(provider, model)` pair, in caller-defined order, within a caller-defined deadline.
How you turn it on	One HTTP header per request: `X-LLM-Fallback-Policy-Id` (named, stored) or `X-LLM-Fallback-Policy` (inline JSON).
What it never does	Auto-apply a default. Cross orgs/projects. Restart a stream after the first byte has been flushed. Swap embeddings models.
What you get back	Three response headers: `X-LLM-Served-By`, `X-LLM-Fallback-Triggered`, `X-LLM-Fallback-Hops`.

When to Use It

High-availability workloads — survive a provider outage by failing over to a second provider hosting the same model (e.g. Anthropic → Bedrock for Claude).
Capacity smoothing — when the primary returns 429s, automatically spill over to a backup target instead of bubbling up the rate limit.
Regional resilience — route around region/zone outages without redeploying.

Quickstart

Store a policy (one-time, per project):

POST /v1/projects/{project_id}/fallback-policies
Content-Type: application/json

{
  "policy_id": "prod-cheap",
  "targets": [
    { "provider": "anthropic", "model": "claude-opus-4-7" },
    { "provider": "bedrock",   "model": "claude-opus-4-7" }
  ],
  "deadline_ms": 15000
}

Reference it on a request:

POST /v1/chat/completions
Authorization: Bearer $LLM_API_KEY
X-LLM-Fallback-Policy-Id: prod-cheap
Content-Type: application/json

{"model": "claude-opus-4-7", "messages": [...]}

Read the result headers:

HTTP/1.1 200 OK
X-LLM-Served-By: bedrock/claude-opus-4-7
X-LLM-Fallback-Triggered: true
X-LLM-Fallback-Hops: 2

That request was attempted on Anthropic first, classified as a trigger error, then succeeded on Bedrock.

Key Concepts

Term	Meaning
Target	A `(provider, model)` pair, plus optional per-hop `timeout_ms`, `max_attempts`, `backoff_ms`.
Policy	An ordered list of 1–8 targets, a chain-wide `deadline_ms`, and a `trigger` describing which errors advance to the next target.
Primary target	The first target in the policy. It is the request's initial attempt; the rest of the chain is the fallback.
Hop	One attempt against one target. The number of hops is reported via `X-LLM-Fallback-Hops`.
Trigger	The rule for advancing to the next target. A hop's error must match `trigger.classes` or `trigger.http_status` — anything else terminates the chain.
Error class	A normalized vocabulary mapping each provider's native error shape to one of `rate_limited`, `server_error`, `timeout`, `auth_error`, etc.
Capability gap	A pre-hop check that skips a target lacking a feature the request needs (e.g. vision). Controlled by `on_capability_gap` = `skip` (default) or `degrade`.

Supported Endpoints

Method	Path	Notes
`POST`	`/v1/chat/completions`	Sync + streaming (pre-first-byte only)
`POST`	`/v1/audio/speech`	Text-to-speech
`POST`	`/v1/audio/transcriptions`	Speech-to-text (HTTP, not WebSocket)
`POST`	`/v1/videos`	Async video generation submit

WARNING

Once a streaming response has flushed its first byte, fallback can no longer restart on a new target — the in-flight stream receives a terminal error event instead.

Enabling Fallback on a Request

Send a policy in one of two ways. Only HTTP headers are accepted — there is no body-embedded transport.

By reference (stored)

Reference a policy you've previously saved on the project. Best for production: centralized management, audit history, shorter request payloads.

POST /v1/chat/completions
X-LLM-Fallback-Policy-Id: prod-cheap

Inline JSON

Pass the full policy JSON in the header (capped at 8 KiB). Best for ad-hoc or programmatic use.

POST /v1/chat/completions
X-LLM-Fallback-Policy: {"targets":[...],"deadline_ms":15000}

INFO

If both headers are present, the inline header (X-LLM-Fallback-Policy) wins. The conflict is recorded on the access log entry (fallback.conflict: true) so it's visible without surfacing as a user-facing error.

Request Validation

The body's model must match the policy's first target. The first target is the request's primary attempt — fallback is the alternate chain that runs only if the primary fails.

WARNING

Sending a policy whose first target points at a different (provider, model) than the body declares is rejected up-front with HTTP 400, before any provider is contacted.

Example — mismatch is rejected:

POST /v1/chat/completions
X-LLM-Fallback-Policy: {"targets":[{"provider":"anthropic","model":"claude-opus-4-7"},
                                   {"provider":"bedrock","model":"claude-opus-4-7"}]}
Content-Type: application/json

{"model": "gpt-4o", "messages": [...]}

HTTP/1.1 400 Bad Request

{
  "error": {
    "message": "request body model \"gpt-4o\" does not match the first target of the fallback policy (\"anthropic/claude-opus-4-7\")",
    "type": "invalid_request_error",
    "param": "model",
    "code": "fallback_policy.body_model_mismatch"
  }
}

Notes:

When model is in provider/model form (e.g. "anthropic/claude-opus-4-7"), both fields are compared.
When model is a bare id (e.g. "claude-opus-4-7"), only the model id is compared.
Validation runs identically whether the policy is supplied inline or by reference.
A rejected request emits a rejected_validation outcome in observability; it never counts as a provider failure or a fallback hop.

Reading the Response

Every response — success or terminal failure — carries:

Header	Meaning
`X-LLM-Served-By`	`provider/model` that actually answered
`X-LLM-Fallback-Triggered`	`true` if a non-primary hop served the response
`X-LLM-Fallback-Hops`	Number of hops attempted

Example — primary succeeded:

X-LLM-Served-By: anthropic/claude-opus-4-7
X-LLM-Fallback-Triggered: false
X-LLM-Fallback-Hops: 1

Example — fallback served the response:

X-LLM-Served-By: bedrock/claude-opus-4-7
X-LLM-Fallback-Triggered: true
X-LLM-Fallback-Hops: 2

Trigger Configuration

A hop's error must match either the policy's trigger.classes set or its trigger.http_status set to advance to the next target. Any other failure (and every success) terminates the chain immediately.

`trigger.classes`

A normalized vocabulary mapped from each provider's error taxonomy. Omit the field entirely to inherit the safe default — request-attributable classes are never in the default and must be listed explicitly.

Default (availability errors — applied when classes is omitted):

Class	Meaning
`rate_limited`	Upstream 429 / quota exhausted / throttled
`server_error`	Upstream 5xx that is not a known capability or routing failure
`region_unavailable`	Region/zone outage or routing-layer failure
`model_unavailable`	Model offline, not deployed in the target region, or temporarily disabled
`timeout`	Per-hop deadline exceeded before headers (or first byte for streams)
`network_error`	TCP/TLS/DNS failure, connection reset, premature EOF

Opt-in only (request-attributable — must be listed explicitly):

Class	When to enable
`auth_error`	Upstream 401/403 — only if you intentionally rotate credentials across providers
`invalid_request`	Upstream 400 with malformed request — only if downstream targets accept different schemas
`context_length_exceeded`	Prompt is too long for this model but smaller for next target
`capability_unsupported`	Feature (e.g. tools, vision) missing on primary but present on fallback
`invalid_response`	Provider returned malformed/unparseable payload
`content_policy`	Provider refused on safety grounds; alternate target may accept

WARNING

The opt-in classes are request-attributable — the failure is caused by your request, not the provider. Enabling them blindly can mask real bugs. Only opt in when you have a concrete reason.

`trigger.http_status`

A list of raw upstream HTTP status codes that also trigger fallback regardless of class mapping. Each value must be 100–599. Use this as an escape hatch when a provider's status doesn't classify cleanly.

Status	Typical cause
`408`	Request timeout
`425`	Too early
`429`	Rate limited
`500`	Internal server error
`502`	Bad gateway
`503`	Service unavailable
`504`	Gateway timeout
`529`	Anthropic overloaded

When Fallback Does Not Apply

Fallback is skipped — and the primary error surfaces — in these cases:

No policy header on the request.
Streaming and the first byte has already flushed → terminal error event on the stream; no restart on a fallback target.
Endpoint is not in the supported list above.
Hop error matches neither trigger.classes nor trigger.http_status.
Chain deadline (deadline_ms) elapsed before the next hop could begin.

Best Practices

Pair the same model across providers for true drop-in fallback (e.g. anthropic/claude-opus-4-7 → bedrock/claude-opus-4-7). Cross-model fallback works but may change response quality or pricing.
Keep deadline_ms realistic. If your client times out at 30s, a chain deadline of 25s is sensible. Too tight, and the second hop never runs.
2–3 targets is the sweet spot. The maximum is 8, but deeper chains amplify the chance that the chain-wide deadline_ms elapses before a healthy hop is reached.
Start with the default trigger classes. Only opt into request-attributable classes (e.g. auth_error, content_policy) when you have a specific reason — they can mask bugs you'd otherwise see.
Use stored policies in production. Inline policies are great for testing, but stored policies give you centralized updates, audit history, and shorter request payloads.
Inspect X-LLM-Fallback-Triggered in your monitoring. A sudden spike means your primary is degrading — fallback is doing its job, but it's also a signal worth alerting on.

FAQ

Can I embed the policy in the request body?

  No. Only the two HTTP headers (`X-LLM-Fallback-Policy-Id` and `X-LLM-Fallback-Policy`) are accepted. Headers are the single uniform transport across every supported endpoint, including the binary/multipart audio and video routes where mutating the body would be costly.

What happens if I send both headers?

The inline header (`X-LLM-Fallback-Policy`) wins. The conflict is recorded on the access log
entry (`fallback.conflict: true`) so it's visible without surfacing as a user-facing error.

Can a policy in project A reference a policy id stored in project B?

No. Stored policies are scoped to a single project; cross-project ids resolve to a 404-shaped
not-found error. This is deliberate — it makes project-membership probing impossible.

Does fallback restart a streaming response after some tokens have been delivered?

No. Once the first byte has crossed to the client, the stream is locked to the current target. A
subsequent upstream failure surfaces as a terminal error event on the existing stream — there is
no second start.

Why isn't /v1/embeddings supported?

Embeddings from different models occupy different vector spaces (and often different
dimensions). Silently swapping the producing model would corrupt the caller's vector store.
Until a "compatible alternates" concept exists, embeddings stay single-shot.

What's the cache TTL on stored policies?

60 seconds, positive **and** negative. Admin mutations publish an invalidation event over Redis
pub/sub so all gateway replicas drop their cached entry immediately — you don't have to wait the
full TTL after a `POST`/`PUT`/`DELETE`.

Is there a hard ceiling on chain depth?

  Yes — 8 targets. In practice 2–3 is the sweet spot; deeper chains amplify the chance that the chain-wide `deadline_ms` elapses before a healthy hop is reached.