Last updated: July 7, 2026
Failover between LLM providers takes one extra field on an OpenAI-compatible endpoint: the Inworld AI LLM Router routes to 220+ models through a single chat completions API and retries a fallback list automatically. In our live test on July 7, 2026, a failing primary model plus a successful Anthropic fallback completed in 1,414 ms total.
Inworld AI is a research lab and inference provider focused on realtime AI models for consumer-facing applications; the
LLM Router is the model-access layer, routing LLM traffic at cost alongside Realtime TTS, Realtime STT, and the Realtime API. This tutorial builds three things with live-tested code: a basic multi-provider completion, provider failover (both Router-native and client-side), and a hash-based A/B test with per-arm cost and latency logging. Every snippet below was executed against the live API on July 7, 2026, and every latency number comes from those runs. If you want background on what a routing layer does before writing code, start with
what an AI router does.
What do you need before starting?
You need one API key and nothing else. The Router speaks the OpenAI chat completions wire format at
https://api.inworld.ai/v1/chat/completions, authenticated with
Authorization: Basic $INWORLD_API_KEY (the key from the
Inworld Portal is already Base64 credentials, so it is Basic, not Bearer). Python needs
requests; the TypeScript examples run on Node 18+ with built-in
fetch, no packages.
# Get your key at platform.inworld.ai (free tier available)
export INWORLD_API_KEY="your-key-here"
If you already use the official OpenAI SDK, it works unchanged: set
base_url="https://api.inworld.ai/v1" and pass your Inworld key as
api_key. We verified this live on July 7, 2026, including failover via the SDK's
extra_body parameter. SDK setup is in the
OpenAI compatibility docs; the router-specific fields that
extra_body can carry are in the
request-level routing docs.
How do you call multiple LLM providers through one API?
Send an OpenAI-format request with a provider/model string. The same endpoint, headers, and response shape work for openai/gpt-5.5, anthropic/claude-sonnet-4-6, google-ai-studio/gemini-3.5-flash, and the rest of the catalog, so switching providers is a one-string change. In our runs this request completed end to end in 928 to 1,617 ms.
# pip install requests
import os
import time
import requests
INWORLD_API_KEY = os.environ["INWORLD_API_KEY"]
URL = "https://api.inworld.ai/v1/chat/completions"
HEADERS = {
"Authorization": f"Basic {INWORLD_API_KEY}",
"Content-Type": "application/json",
}
t0 = time.perf_counter()
response = requests.post(URL, headers=HEADERS, json={
"model": "openai/gpt-5.5",
"messages": [{"role": "user", "content": "Say hello in one sentence."}],
}, timeout=30)
response.raise_for_status()
data = response.json()
print(data["choices"][0]["message"]["content"])
print(f"end-to-end latency: {(time.perf_counter() - t0) * 1000:.0f}ms")
print(data["usage"])
The response is standard OpenAI shape plus a metadata block that the failover and A/B sections below rely on. This is the actual response from one of our test runs (key values unmodified):
{
"id": "chatcmpl-1783480163360",
"object": "chat.completion",
"created": 1783480163,
"model": "openai/gpt-5.5",
"choices": [
{
"index": 0,
"finish_reason": "stop",
"message": { "role": "assistant", "content": "Hello!" }
}
],
"usage": { "prompt_tokens": 12, "completion_tokens": 5, "total_tokens": 17 },
"metadata": {
"attempts": [
{ "model": "openai/gpt-5.5", "time_to_first_token_ms": 641, "success": true }
],
"generation_id": "121aa8e2-5e6d-4223-a013-99d16e834c67",
"total_duration_ms": 647
}
}
Across our July 7, 2026 test runs, openai/gpt-5.5 time-to-first-token as reported in metadata.attempts ranged from 641 to 1,936 ms on short prompts. These are single-run illustrative numbers from one client location, not a benchmark; run your own measurements from your production region.
How does failover between LLM providers work?
Two ways: let the Router do it, or do it in your client. The Router natively retries a per-request fallback chain on any provider error and can also fall back when the first token is slower than a latency budget you set. Client-side loops still matter for conditions the Router cannot see, like your own content checks. The table summarizes what we verified.
The native chain is one field. Put backups in models, in priority order:
import os
import requests
INWORLD_API_KEY = os.environ["INWORLD_API_KEY"]
URL = "https://api.inworld.ai/v1/chat/completions"
HEADERS = {
"Authorization": f"Basic {INWORLD_API_KEY}",
"Content-Type": "application/json",
}
response = requests.post(URL, headers=HEADERS, json={
"model": "openai/gpt-5.5",
"messages": [{"role": "user", "content": "Reply with the single word: ok"}],
# tried in order if the primary model errors out
"models": ["anthropic/claude-sonnet-4-6", "google-ai-studio/gemini-3.5-flash"],
}, timeout=30)
response.raise_for_status()
data = response.json()
print("served by:", data["model"])
for attempt in data["metadata"]["attempts"]:
print(attempt)
To watch it actually fail over, we sent a request whose primary was an invalid model ID with anthropic/claude-sonnet-4-6 as fallback. The Router rejected the primary, moved down the chain, and returned HTTP 200 in 1,414 ms total. The metadata.attempts array is the audit trail (real output, error string truncated):
[
{
"model": "openai/gpt-nonexistent-model",
"status_code": 400,
"error": "rpc error: code = InvalidArgument desc = The requested model 'gpt-nonexistent-model' is currently not supported. ...",
"success": false
},
{
"model": "anthropic/claude-sonnet-4-6",
"time_to_first_token_ms": 1229,
"success": true,
"credential_type": "system"
}
]
The top-level model in the response is the model that actually answered (anthropic/claude-sonnet-4-6 here), so log data["model"], not the model you requested.
For latency-triggered failover, add
"fallback": {"ttft_timeout": "1000ms"} to the request body (
request-level routing docs). In our live test with a 1,000 ms budget, the Router abandoned
openai/gpt-5.5 at exactly 1,000 ms with the warning
TTFT timeout exceeded; fell back to next model and served the reply from
google-ai-studio/gemini-3.5-flash (872 ms TTFT, 2,097 ms total). One sharp edge we hit: the budget applies to every attempt, so if no model in your chain can meet it, the whole request fails. With a 500 ms budget, when our measured frontier TTFTs ran 640 to 1,940 ms, we got HTTP 502 back in about 1.3 s. Set the budget above the typical TTFT of your fastest fallback.
Get an API key and run the failover example in minutes: create a free key at
platform.inworld.ai, export it, and paste either snippet above. The same key covers the full catalog.
Should you build failover client-side instead?
Use client-side failover when the trigger is something only your application can judge: your own end-to-end deadline, a response that fails validation, or redundancy across two independent gateways. The pattern is a loop over candidates with a per-attempt timeout. It costs you the elapsed time of every failed attempt, so budget accordingly.
import os
import requests
INWORLD_API_KEY = os.environ["INWORLD_API_KEY"]
URL = "https://api.inworld.ai/v1/chat/completions"
HEADERS = {
"Authorization": f"Basic {INWORLD_API_KEY}",
"Content-Type": "application/json",
}
def complete_with_failover(messages, candidates, timeout_s=8.0):
last_error = None
for model in candidates:
try:
r = requests.post(URL, headers=HEADERS,
json={"model": model, "messages": messages},
timeout=timeout_s)
r.raise_for_status()
return r.json()
except (requests.Timeout, requests.HTTPError,
requests.ConnectionError) as err:
last_error = err
print(f"{model} failed ({type(err).__name__}), trying next model")
raise last_error
data = complete_with_failover(
[{"role": "user", "content": "One word: what color is grass?"}],
["openai/gpt-5.5", "google-ai-studio/gemini-3.5-flash"],
)
print("served by:", data["model"])
We verified the failure path by shrinking the timeout to 0.8 s, below gpt-5.5's typical first-token time in our runs: the loop logged openai/gpt-5.5 failed (ReadTimeout), trying next model and moved on exactly as intended. Two caveats. First, worst-case latency is roughly timeout × number_of_candidates, so a chatty 8 s timeout across three candidates can hold a user for 24 s; the Router-native chain avoids most of that because upstream errors surface in hundreds of milliseconds rather than waiting out your client timeout. Second, a client loop through one gateway still shares that gateway's fate. If you need gateway-level redundancy, run this same loop across two independent base URLs. That is a real architectural advantage of the client-side pattern, and no single vendor, Inworld included, can give it to you natively.
How do you A/B test LLM models on live traffic?
Hash a stable user ID into 100 buckets and map bucket ranges to models. The split is deterministic, so users never flip arms mid-experience, and no assignment database is needed. Because arms only differ by a model string on one API, the experiment logic is a hash function plus a lookup table. Log model, latency, and token usage per request.
import hashlib
import json
import os
import time
import requests
INWORLD_API_KEY = os.environ["INWORLD_API_KEY"]
URL = "https://api.inworld.ai/v1/chat/completions"
HEADERS = {
"Authorization": f"Basic {INWORLD_API_KEY}",
"Content-Type": "application/json",
}
ARMS = {"A": "openai/gpt-5.5", "B": "anthropic/claude-sonnet-4-6"}
def assign_arm(user_id: str, split: int = 50) -> str:
"""Deterministic 50/50 split. Same user always lands in the same arm."""
bucket = int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 100
return "A" if bucket < split else "B"
def chat(user_id: str, content: str):
arm = assign_arm(user_id)
t0 = time.perf_counter()
r = requests.post(URL, headers=HEADERS, json={
"model": ARMS[arm],
"messages": [{"role": "user", "content": content}],
"user": user_id, # stable end-user id; also enables sticky managed routing
}, timeout=30)
r.raise_for_status()
data = r.json()
usage = data["usage"]
log_line = {
"user": user_id,
"arm": arm,
"model": data["model"],
"latency_ms": round((time.perf_counter() - t0) * 1000),
"ttft_ms": data["metadata"]["attempts"][-1].get("time_to_first_token_ms"),
"prompt_tokens": usage["prompt_tokens"],
"completion_tokens": usage["completion_tokens"],
"cached_tokens": usage.get("prompt_tokens_details", {}).get("cached_tokens", 0),
"generation_id": data["metadata"]["generation_id"],
}
print(json.dumps(log_line)) # ship this to your analytics pipeline
return data
for uid in ["user-1001", "user-1002", "user-1003", "user-1004"]:
chat(uid, "Suggest one healthy breakfast in one sentence.")
We ran this split across six synthetic users on July 7, 2026. Assignment was identical between the Python and Node implementations (SHA-256 is SHA-256), and the per-arm logs looked like this:
Single-run numbers on an identical short prompt, shown to illustrate the logging, not to rank the models. One immediately useful signal even at this tiny scale: arm B consistently produced 42 to 47 completion tokens against arm A's 25 to 27 on the same question, and completion tokens are the expensive ones, so verbosity differences show up directly in cost per request.
If you would rather not own the assignment code, the Router does this as managed configuration: create a router object via
POST https://api.inworld.ai/router/v1/routers with weighted variants (weights must sum to exactly 100 per route), reference it as
"model": "inworld/your-router-name", and pass a stable
user field for sticky per-user assignment, per the
traffic splitting docs. That moves split changes out of your deploy cycle: shifting 10% to 50% is a config edit, not a code release. We did not exercise the managed path in this tutorial; the code path above is fully live-verified.
How do cached tokens show up in the usage field?
The Router passes through provider cache accounting. When a repeated prompt prefix hits a provider-side prompt cache, usage.prompt_tokens_details.cached_tokens appears in the response, and those tokens are billed at the provider's cached-input discount. If your A/B arms carry a long shared system prompt, per-arm cost math that ignores this field will overstate spend.
We measured it directly: three identical requests with a ~6,400-token system prompt to openai/gpt-5.5, about one second apart (July 7, 2026). Call one paid full freight; calls two and three reported 5,888 of 6,431 prompt tokens as cached:
{
"completion_tokens": 30,
"completion_tokens_details": { "reasoning_tokens": 20 },
"prompt_tokens": 6431,
"prompt_tokens_details": { "cached_tokens": 5888 },
"total_tokens": 6461
}
The fields worth logging on every request, with the values we actually observed:
For cost attribution, multiply per-arm token counts by list rates and apply the cache discount to
cached_tokens. As a dated anchor: OpenAI lists gpt-5.5 at $5.00 per 1M input tokens and $30.00 per 1M output tokens (developers.openai.com pricing, fetched July 7, 2026). The Router itself adds no markup; routing is at cost across 220+ models per the
Inworld pricing page, as of July 7, 2026.
When is a different tool the better choice?
A router is not always the right layer, and Inworld's is not the only good one. Four cases where something else wins:
- You need routing logic inside your own VPC. A self-hosted proxy like LiteLLM keeps every routing decision and log in your infrastructure, at the price of operating it yourself. See our comparison of LiteLLM alternatives for the trade-offs.
- You want a bring-your-own-key marketplace and its ecosystem. OpenRouter has a large catalog and a mature BYOK model. We wrote up the differences, including where OpenRouter wins, in Inworld Router vs OpenRouter.
- You use exactly one provider and its exclusive surfaces. If you depend on provider-specific features like OpenAI's Realtime API voice sessions or a provider's batch and fine-tuning endpoints, the native SDK is the shorter, better-supported path. A router earns its hop when you use two or more providers.
- You need gateway-level redundancy. Any single gateway, Inworld included, is one more dependency in your request path. The client-side loop above pointed at two independent endpoints is the honest answer, not any vendor's native failover.
For a broader survey of the category, see the
best LLM routers and AI gateways.
Production ops checklist
Working failover and clean experiment data come from the same eight disciplines:
- Set explicit timeout budgets end to end. Client timeout > Router TTFT budget > typical model TTFT. Our measured frontier-model TTFTs ran 641 to 1,936 ms on short prompts (July 7, 2026), so a 300 to 500 ms TTFT budget starves every attempt; we drew a 502 at 500 ms.
- Retry on a different provider, not the same one. A provider incident or 429 usually fails identically on immediate same-provider retry. The
models chain switches providers by construction.
- Keep fallback arms behaviorally compatible. Same context-window class, same tool-calling support, and a system prompt tested against every model in the chain, or your fallback trades an outage for a quality incident.
- Make assignment sticky on a stable ID. Hash the durable user ID, never a session or request ID, and pass it in the
user field so managed routing stays consistent with your client-side split.
- Log what actually happened, not what you requested. Under failover, response
model differs from the requested model. Log model, arm, end-to-end latency, per-attempt TTFT from metadata.attempts, and generation_id for tracing.
- Account for cache effects in per-arm cost. Include
cached_tokens in the cost formula; a long shared system prompt can sharply cut the cost of second-and-later requests, as our 5,888-of-6,431 measurement shows.
- Alert on fallback rate, not just error rate. Native failover hides provider incidents from your error dashboards by design. A rising average length of
metadata.attempts is your early warning.
- Decide the end-of-chain behavior. When every model fails, the Router returns an error (we observed 502). Catch it and serve a graceful degradation, such as a canned response or a retry-later message.
About Inworld AI
Inworld is a research lab and inference provider focused on realtime AI models for consumer-facing applications. We build first-party voice models (Realtime TTS and Realtime STT), serve optimized open-source LLMs on our own Realtime Inference engine, and expose them as modular APIs, alongside an LLM Router that routes to 220+ models and a Realtime API for full speech-to-text-to-LLM-to-speech pipelines. We focus on serving developers of realtime, high-volume conversational products across domains such as health, fitness, education, companions, social, and games, with an emphasis on quality, low latency, and low cost at scale.
Frequently asked questions
How do I add failover between LLM providers with an OpenAI-compatible API?
Add a fallback list to the request body. Through the Inworld Router, POST to /v1/chat/completions with your primary in the model field and backups in a models array, for example anthropic/claude-sonnet-4-6 after openai/gpt-5.5. If the primary errors, the Router tries each fallback in order and records every attempt in the response metadata.attempts array. No client-side retry code is required.
Does the OpenAI SDK work with the Inworld Router for failover?
Yes. Point the official OpenAI SDK at base_url="https://api.inworld.ai/v1" with your Inworld API key and existing chat-completions code runs unchanged; we verified this live on July 7, 2026. Pass the fallback list through the SDK's extra_body parameter, for example extra_body={"models": ["anthropic/claude-sonnet-4-6"]}. In our test, a request with an invalid primary model was answered by the Anthropic fallback.
How do I A/B test two LLM models without changing my API integration?
Hash a stable user ID into buckets (for example SHA-256 mod 100) and map buckets to models, so each user deterministically lands in one arm. Because the Router is one endpoint for 220+ models, switching arms only changes the model string. Log arm, model, latency, and the usage token counts per request. The Router also supports managed weighted
traffic splitting with sticky per-user assignment via the
user field.
What happens when every fallback model fails?
The Router returns an error after exhausting the chain. In our July 7, 2026 tests, setting a time-to-first-token budget that no model in the chain could meet (500 ms, against measured frontier-model TTFTs of 641 to 1,936 ms) returned HTTP 502 in about 1.3 seconds. Your client should catch that final error and degrade gracefully, for example with a cached response or an explicit retry-later message.
Does the Router report cached tokens in the usage field?
Yes, when the upstream provider reports them. On a repeated request with a ~6,400-token system prompt to openai/gpt-5.5, the second call returned usage.prompt_tokens_details.cached_tokens of 5,888 out of 6,431 prompt tokens (measured July 7, 2026). Reasoning models also surface completion_tokens_details.reasoning_tokens. Log both per A/B arm, because cached tokens are billed at a provider discount and change each arm's effective cost.
How much does the Inworld LLM Router cost?
Routing is at cost: you pay the underlying provider token rates with no markup, across 220+ models, per the
Inworld pricing page as of July 7, 2026. Failover attempts rejected upstream before any generation, such as an invalid model or a provider error, do not add token cost; you pay for the model that actually answers. Signup is free and the same API key also works for Inworld
Realtime TTS and STT.
Next steps