How to Host Open-Source LLMs (Gemma, gpt-oss, DeepSeek) for Production Traffic in 2026

What is the best way to host open-source LLMs in production?

1. Self-host on cloud GPUs. Rent H100, H200, or B200 instances from AWS, GCP, Azure, Oracle, CoreWeave, Lambda, or a neocloud. Run vLLM, SGLang, or TensorRT-LLM yourself. Maximum control, maximum engineering ownership.
2. Managed hosting. Use providers like Fireworks, Cerebras, Together.ai, DeepInfra, Modal, or SambaNova. They run the inference stack; you call an API. Per-token or per-second billing.
3. Routing layer with 1P-optimized open-source models. Use a router that exposes both 3P providers and a first-party track of self-optimized open-source models. Inworld Realtime Router is the example. One API, OpenAI SDK compatible, with the 1P track tuned for cache-friendly production workloads.

Why are developers moving off pure self-hosting?

• GPU supply. B200 capacity is tight in 2026. Multi-month commitments are common.
• Optimization gap. Out-of-box vLLM throughput is often 3 to 4x below a tuned deployment. FlashInfer kernels, speculative decoding, and NVFP4 quantization are where the wins live.
• Variable traffic. Consumer apps spike. Reserved capacity priced for peak burns money at trough.
• Tail latency. Sub-second TTFT on cache-friendly workloads requires ongoing tuning, not a one-time deploy.

How do hosted, serverless, and dedicated GPU options compare?

Which open-source LLMs are production-ready in 2026?

How does managed hosting break down across providers?

How does Inworld optimize hosted open-source models?

• vLLM as the serving engine, with a custom FlashInfer kernel patch (flashinfer-ai PR #2959).
• Speculative decoding with model-specific draft pairings.
• KV cache optimization for cache-friendly input-heavy workloads.
• NVFP4 quantization on NVIDIA B200 GPUs.
• Custom kernels for the high-throughput paths.

Where does first-party hosting beat aggregation?

• Voice agents. When the LLM call sits between STT and TTS in a single conversation, a shared auth and inference fabric removes a network hop and removes provider-mismatch tail latency. Inworld Realtime API does this with the Router as the LLM layer.
• High cache-hit-rate workloads. Character chat and roleplay apps reuse system prompts and persona context across requests. Janitor AI runs approximately 600B tokens per day on Inworld-hosted fine-tuned Gemma with cache-hit-rate as a core operational metric.
• Long-session consumer apps. Latitude runs DeepSeek V3.2 on a dedicated Inworld cluster for AI Game Master sessions. Yonder runs the same V3.2 cluster for its workload. AI Roguelite migrated Gemma 4 31B traffic and reports it outperforming DeepSeek-reasoning for their case.

How do I call Inworld 1P open-source models via the Realtime Router?

import os
import requests
import base64

api_key = os.environ["INWORLD_API_KEY"]
auth = "Basic " + base64.b64encode(api_key.encode()).decode()

# Inworld Realtime Router, OpenAI Chat Completions compatible
# 3P open-source via DeepInfra; for 1P Realtime Inference use inworld/ prefix (e.g. inworld/gemma-4-26b)
response = requests.post(
    "https://api.inworld.ai/v1/chat/completions",
    headers={
        "Authorization": auth,
        "Content-Type": "application/json",
    },
    json={
        "model": "deepinfra/openai/gpt-oss-120b",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Summarize hosted vLLM in one sentence."},
        ],
    },
    timeout=60,
)
data = response.json()
print(data["choices"][0]["message"]["content"])

json={
    "model": "deepseek/deepseek-v4-pro",
    "messages": [...],
}

from openai import OpenAI

client = OpenAI(
    base_url="https://api.inworld.ai/v1",
    api_key=os.environ["INWORLD_API_KEY"],
    default_headers={"Authorization": auth},  # Basic auth, NOT Bearer
)

completion = client.chat.completions.create(
    model="deepseek/deepseek-v4-pro",
    messages=[{"role": "user", "content": "Explain speculative decoding briefly."}],
)
print(completion.choices[0].message.content)

What about voice in the same stack?

How should I decide between the three paths?

• Pick self-hosting when sovereignty rules out third-party inference, when load is steady and reserved capacity makes sense, and when you already have an inference engineering team.
• Pick managed hosting when you want to start in a day, when traffic is variable, when you are A/B testing across many models, or when you need cost-focused per-token billing on a broad catalog. Fireworks for fine-tune-plus-serve, Cerebras for raw token-per-second on supported models, Together for mixed serverless and clusters, DeepInfra for cost, Modal for Python-native custom stacks.
• Pick Realtime Inference (the 1P track of Realtime Router) when you want optimized hosted Gemma 4, DeepSeek V3.2/V4, or MiniMax-M2.5 in the same API that already routes across OpenAI, Anthropic, Google, Mistral, Groq, Fireworks, and DeepInfra (with gpt-oss-120b available via DeepInfra on the 3P track), especially when voice is in the pipeline. Capacity on the 1P track is allocated to production workloads. Contact for current availability.

FAQ

What is the best way to host open-source LLMs in production?

Which open-source LLMs are production-ready in 2026?

How does Inworld optimize hosted open-source models?

When should I self-host instead of using a managed provider?

Can I use the OpenAI SDK with Inworld Realtime Router?

How does Inworld Realtime Router compare to Fireworks, Cerebras, Together, DeepInfra, and Modal?