Introducing Inworld Inference

01Consumer vs. enterprise inference

For an enterprise task that replaces expensive manual work, a single result can justify a lot of compute, and the result can wait. A consumer app has to deliver interaction after interaction inside one subscription or one free session, while a person waits for each reply. So the number to optimize is the cost of one completed interaction, at the quality and response time the product needs.

Our goal is to get this down to a few cents per user per day, and that's something that you can only do with true experts and an ML team that really understands the trade-offs and techniques for accomplishing that.

FaiStatus by Wishroll

The table below sets the two workloads side by side, with what we built for each.

The tightest constraint is time. In conversation, people leave 0 to 300 ms between turns.^[7] The model's first token is only part of that window, because the network and, in a voice app, speech synthesis use the rest. So the first token has to arrive well inside 300 ms, at production load. On Gemma 4 26B, the median first token now lands in 117 ms, which leaves most of the window for the voice.

The second difference is the size of each request. A consumer turn sends the whole conversation as the prompt and gets one short reply back. Across the apps we served in the 28 days to 23 September, median prompts were 13 to 36 times the length of replies. Enterprise work leans the other way, toward long outputs such as code and documents.

That workload already runs on Inworld at scale. On the most-used Gemma 4 models, we process up to 1.8 times the daily tokens that OpenRouter publicly reports across every provider on its platform combined.

We also serve those models faster. OpenRouter publishes first-token latency for every provider of a model. Against those measurements:^[6]

• Gemma 4 26B: the lowest median (0.17 s) and p95 (0.31 s) first token of the providers compared. That is 3.6 times faster at the median and 7.8 times faster at p95 than the median provider, at an input price nearly 40% below the median provider's.
• MiMo 2.6 Pro: the lowest input price of the providers on OpenRouter, with the lowest median (0.39 s) and p95 (0.91 s) first token. That is 2.6 times faster at the median and 4.6 times faster at p95 than the next-fastest provider.

The sections below cover the calls a consumer app makes, then each technique and what it changes for the app.

02Consumer inference tasks, and how we serve each

A consumer app doesn't make one kind of model call. A single turn can check the message for safety, classify what the person wants, read their memory, write the reply, call a tool and update the memory afterwards. Each of those calls has a different size, a different deadline and a different budget, and a provider built around one of them leaves the rest slow or expensive.

The same work sits underneath every row: kernels and decoding built for each model, a router that keeps each conversation's cache warm, precision checked against the evals, and a service tier per request so each call gets the speed or price it needs.

03The serving challenge: long context in, short reply out

Consumer apps have a distinctive traffic pattern. They send very long context to get a short reply, so processing and reusing the input matters as much as generation speed. And the calls one user action triggers each have their own size and deadline, as the timeline below shows.

A dialogue turn with 3,500 tokens of history and a 200-token reply puts most of the work on the prompt. Long conversations pull the average far above the median: on GLM 5.3 Flash, the average request is 13,433 tokens in for 309 out.

A story with a longer answer shifts the weight to decoding. The two phases hit different limits. Prefill, the work of processing the input, is bound by compute and gains from cache reuse and batching. Decode, the work of generating the output, is bound by memory bandwidth and gets faster when each step is cheaper or produces more than one token.

So we work on both sides of the turn. On the prompt side, we avoid recomputing context the model has already seen, and chunked prefill processes a long prompt in pieces so it never holds up the replies streaming to other users. On the output side, we make every decode step cheaper and let each step produce several tokens. And because a background summary can wait while a reply cannot, every request carries a service tier. The next two sections cover the prompt side first, then the output side.

04KV-aware routing for long conversations at scale

Each turn resends the conversation so far. Prefix caching keeps the attention keys and values (the KV cache) for context the model has already processed, so the engine computes only the new part of the prompt. Every reply is still generated fresh.^[2]

At fleet scale, the hard part is placement. A conversation's cache lives on the replica that served its last turn, while another replica may have more free capacity. Our router is KV-aware. It hashes each block of the prompt, tracks which replica holds which blocks, and scores every replica on two things: how much of this request's prefix it already holds, and how busy it is. Sending every turn to the warmest cache would overload it, so the score trades reuse against load, tuned on representative traffic. KV that doesn't fit in accelerator memory spills to host memory instead of being evicted and recomputed.

Built into each engine, model by model

Cache-hit rate is one of the numbers we watch most closely, because it sets both how soon a reply starts and what the turn costs. Getting it high on real traffic takes more than turning prefix caching on. Different models and engines represent cached context differently: draft modes for realtime decoding, hybrid attention layouts, recurrent state and data-parallel ranks each change how prompt tokens become cache blocks, and the router finds a match only when its view agrees with the engine's.

So we integrate the router with every model's serving path and measure the hit rate on production traffic. That work roughly doubled cache hits on GLM 5.3 Flash, from 25% to 47% on a replay of 5,000 production requests, and tripled them on DeepSeek V4.1 Flash. Our stack holds high hit rates on mixed traffic from many apps at once, and on one customer's dedicated deployment, cache hits reach 80% in production.^[1]

For your app. MiMo 2.6 Pro lists cached input at $0.0036 per million tokens and uncached input at $0.425, so every cached token costs 99% less. When 90% of a prompt comes from cache, the input cost of that turn falls by about 89%, and only a tenth of the prompt has to be processed before the reply can start.

Under the hood. Each cached block skips its share of prefill. With more of every prompt coming from cache, each replica can carry more conversations at once, and time to first token grows far more slowly as a conversation gets longer.

05Faster replies through model-specific decoding and precision

People experience latency twice: while waiting for a response to begin, and while waiting for it to finish. Time to first token (TTFT) measures the first wait. Inter-token latency (ITL) sets the pace of the stream. A serving configuration has to handle both at the app's production concurrency.

Draft-and-verify, tuned for each model

Realtime decoding is our draft-and-verify serving approach, based on speculative decoding. A lightweight draft model proposes several tokens, and the target model verifies them together in one forward pass, accepting consecutive tokens until the first rejection. With exact acceptance and correction, generation can advance by multiple tokens per target-model step while preserving the target model's output distribution, with the same quality.^[3]

We optimize the drafts and the verify step for each model. Gemma 4 uses the draft models Google publishes for it, and GLM and DeepSeek use their own multi-token prediction layers. On a dedicated deployment, with the customer's approval, we train the draft on a sample of that app's traffic. A draft that predicts that app's outputs more accurately gets more tokens accepted per verification step, and we measure the gain at production load, because draft overhead, acceptance rate and concurrency all affect it.

Verify attention on split-KV: up to 7.5× faster

On Gemma 4 26B, a mixture-of-experts model, the verify step turned out to be the bottleneck. Verifying a 7-token draft sends 8 query tokens per sequence through attention, and the stock kernel only uses its faster split-KV path for single-token decoding. Verify fell back to a 2D kernel that split the work into about 60 blocks and left most of the chip's compute units idle.

We changed the kernel so verify batches of up to 16 query tokens take the split-KV path. On the model's global (full-context) attention layers, verify attention ran 3.4 to 7.5 times faster, and attention fell from 27.5% to 16.5% of each decode step. The full build, which also runs the whole verify step as one captured graph and sets the draft length to 7 tokens, took ITL p50 from 6.4 to 5.2 ms and TTFT p50 from 154 to 117 ms on replayed production traffic, with the same math and reasoning scores.^[1]

For your app. ITL sets how fast a reply streams: at 5.2 ms per token, a 200-token reply finishes in about a second after it starts. Faster decode steps also let each accelerator serve more requests, which is what keeps prices low.

Under the hood. Decode is bound by memory bandwidth: every step reads all of the weights to produce one token per sequence. A draft lets one read of the weights produce several tokens, so the speed of the verify step sets the gain. When attention dropped from 27.5% to 16.5% of the step, every request in the batch got faster.

Same eval scores, fewer bytes per token

A cheaper build ships only if it scores the same. Every build has to match the production build it replaces on our evals, including MATH, MMLU-Pro and GSM8K. Within that gate, each layer runs at the lowest precision its evals allow. On Gemma 4 31B, the MLP, which holds most of the weights, runs in 4-bit floating point (NVFP4), while attention and the KV cache stay at FP8.

The gate catches regressions before they ship. It found a scaling bug in FP8 attention, and the fix brought MATH back from 87.4 to 89.6. It also blocked a build that was 80% cheaper to serve, because that build lost 5 points on MATH.

For your app. Fewer bytes per weight keeps inference affordable as your app grows, at the same latency and quality. That is part of how Gemma 4 26B lists at $0.07 per million input tokens and $0.34 output.

06Continuous optimization, model by model

Our work doesn't stop when a model goes live. We continually profile production traffic to improve response times, serving efficiency and reliability, from new attention kernels that accelerate generation to resolving memory leaks that require periodic restarts. Each change is evaluated against our quality benchmarks before release. The table below shows a selection of these improvements and their measured results.

Measured serving improvements

A selection of changes from across the models we host, by kind of work. Each result is before and after on the named model and metric; the basis column says how and when it was measured. Replays use real production traffic.^[1]

Tracking the effect in production

The gains hold in production. On Gemma 4 26B, each serving instance now handles about 1.8 times the requests it did in September, while 7-day median TTFT fell from 178 to 161 ms. On Gemma 4 31B, one week of optimization made median TTFT 2.8 times faster on replayed production prompts, from 943 to 336 ms at the same eval scores, and in production the daily median fell from 365 to 271 ms even as traffic rose 26%. In a replay, cache-aware routing on DeepSeek V4 Flash made the slowest 1% of first tokens about 10 times faster, from 9.4 s to 0.87 s.^[1]

07Flexible service tiers and strong reliability

A reply someone is waiting for needs a fast response; a background summary can wait and prioritize lower cost. Our four serving tiers let you choose the right balance for each request, using the same model weights. Set service_tier per request, or choose a default for your API key or workspace.

Serving choices for the same model weights. Availability and commitments depend on the model and service.

Reliable serving is a priority across every tier. Over 30 days, GLM 5.3 Flash served more than 20 million requests with 98 failures, a 99.9995% success rate, counting server errors, timeouts and capacity rejections. Across all hosted models, more than 99.99% of requests completed without a server error; that fleet-wide metric counts server errors only.^[1]

At about 4.8 failures per million, someone making 100 requests a day would see about one failure every six years, counted by the final status returned to your app. When a serving pool is saturated or unavailable, the router can redirect requests to another pool, helping keep your app responsive without requiring you to manage recovery between serving pools.

08Routing: failover, conditions and A/B splits

Configure a primary model and compatible fallbacks once, so your app can recover without building its own model-switching logic. List fallback models in models and set fallback.ttft_timeout to specify how long to wait for the primary model's first token. If it misses that deadline, the router tries the next model on the list. A saved router can also route on the request's metadata or messages with conditions written in CEL (Common Expression Language), split traffic between variants by weight for an A/B test, and keep each user on one variant with user. When the models in a route are ones we host, each model tried keeps the caching and realtime decoding described above.

Every response says what happened. metadata.attempts lists each model tried, its time to first token and whether it succeeded, and metadata.reasoning says why the route went where it did, so you can debug a routing decision from the response alone.

Because the router treats a slow first token as a failure, a stalled model costs the turn the timeout plus the fallback's first token (300 + 157 ms in the trace above) instead of however long the first model would have taken, and your app needs no retry logic.

09Your model, optimized for your app

Start with hosted open models and choose a serving tier for each task. When your app needs its own model, the goal is a better agent for your users at lower cost and lower latency, and we build it with you on a dedicated deployment.

Bring your own weights, or work with our engineers on post-training with an approved dataset and app-specific evaluations. We measure quality against the model you run today, then tune for efficiency until the new model meets your quality targets for less. Post-training changes what the model does; a draft trained on your traffic changes only how fast we serve it, so we evaluate the two separately.

The deployment can also include custom kernels, cache routing tuned to your traffic and capacity matched to it. On Gemma 4 31B, our own attention kernel serves about 4 times the tokens a second of the stock kernel at full load, and one customer's 31B fine-tune gained 35% throughput when we converted it to 4-bit weights. After launch, we keep profiling the deployment on your traffic, and a change ships only when it holds its gains at production load.

10Customers: Status and Latitude

Customer resultsfrom their case studies

>95%lower AI costs, with content quality kept or improved19 daysto a million usersTeam of fivebuilt on Inworld

Tens of millionsof inference calls in six weeks, about 20 per turn1 dayto the first production task1 weekto half of engine traffic~1 monthto all story-model traffic

For Status by Wishroll, we worked with the team on targeted prompts, app-specific evaluators and production load testing, and the new setup cut their AI costs by more than 95% while keeping or improving content quality. The saving came from that work as well as from a lower token price.^[4]

The real upside with Inworld is that it's a continued relationship that allows us to keep optimizing performance and also keep lowering the costs because now we have this team of ML experts working with us.

FaiStatus by Wishroll

Latitude's Voyage makes many calls per turn. Roughly 20 AI tasks run each turn alongside story generation, which added up to tens of millions of inference calls in Latitude's first six weeks on Inworld. We serve the World Engine tasks, the calls that apply the story's rules each turn, and the story models, including Latitude's own fine-tunes. The team moved its first production inference task in a day, half of its engine traffic within a week, and all of its story-model traffic in about a month. A user waits on the combined workflow, so every supporting call has to finish inside the turn.^[5]

11Get started with the OpenAI-compatible API

For Chat Completions, set the Inworld base URL, your API key and a model ID. Streaming works the same way it does with OpenAI.

import os
from openai import OpenAI

api_key = os.environ["INWORLD_API_KEY"]

client = OpenAI(
    base_url="https://api.inworld.ai/v1",
    api_key=api_key,
    default_headers={"Authorization": f"Basic {api_key}"},
)

stream = client.chat.completions.create(
    model="inworld/models/gemma-4-26b-a4b-it",
    messages=[
        {
            "role": "system",
            "content": "You reply to users of a running app. "
                       "Keep replies short and encouraging.",
        },
        {
            "role": "user",
            "content": "I finally finished my first 10k run today.",
        },
    ],
    max_tokens=128,
    stream=True,
)

with stream:
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
print()

Gemma 4 26B lists at $0.07 per million input tokens, $0.035 cached and $0.34 output, nearly 40% below the median provider that doesn't train on your data. These are Default rates; lower rates are available as you scale. Every hosted model, with its price and cached-input rate, is on the models page. You can try any model in the browser first, with no API key, in the playground.

Browse models · Get an API key · Talk to our engineers about a deployment for your app.