The Vision
Every builder in conversational AI shares a common goal: to create systems that feel natural, responsive, and personalized. But in practice, we spend more time wiring APIs, debugging, and optimizing latency rather than optimizing user experience.
Inworld Reflection
We started at Inworld by building lifelike AI characters that gamers loved—ones that could remember, converse naturally, and feel real.
As our customer base expanded beyond games, they asked for complex customizations—to plug in their own models, connect to proprietary data, define custom emotions, routing, and more.
With each request, our engineering teams spent less time on shipping user features and more time writing integrations and debugging.
This realization led us to a critical analysis of where our development time was truly going.That analysis revealed three recurring engineering pain points in building realtime conversational AI, and we built
Inworld Runtime to solve them.
Inworld Runtime
Inworld Runtime is a low-latency AI backend for realtime conversational AI. You build your conversational AI with Inworld Runtime SDK, launch a hosted endpoint using Inworld CLI, and observe and optimize your conversational AI by running A/B experiments in the Inworld
Portal.
The 3 Challenges of Realtime Conversational AI
Problem 1: Latency that breaks the realtime feel
Before: High latency under high loads
- Scaling issues: As apps scaled to thousands of users, latency spiked above one second.
- Blocking operations: Many popular programming languages, while excellent for rapid prototyping, have runtime limitations that prevent true parallel execution, leading to blocked operations when we need to run multiple LLM calls, embeddings, and processing tasks concurrently.
With Runtime: True parallel execution at the C++ core
- Parallel execution: Using Runtime, an agent can embed user input, retrieve knowledge, and do a web search all at once, then proceed to the LLM call, dramatically reducing end-to-end latency.
- Pre-optimized backend: The graph executor automatically identifies nodes without dependencies and schedules them in parallel — no manual threading code required.
- Read Streamlabs case study: Built a realtime multimodal streaming assistant with sub-500 millisecond latency
Problem 2: 50% dev time spent in integration and debugging
Before: Repetitive, time-consuming tasks
- Wrote repetitive integration code: For every new feature that required integrating an AI model, we found ourselves writing similar integration code.
- Reconstructed execution paths by hand: When an agent's behavior was incorrect, our primary tool for analysis was traditional logging. We had to sift through disconnected logs from various parts of the codebase to manually reconstruct the sequence of events.
- Coupled orchestration and business logic: The control flow for handling model responses, error retries, and feature-specific logic—like updating how fallback responses were triggered—was deeply embedded within the business logic, making even minor feature updates risky. Bringing new developers up to speed took weeks instead of days.
With Runtime: Less Maintenance, More Iteration
- Build fast with pre-optimized nodes: Developers get a full suite of nodes to construct realtime AI pipelines that can scale to millions of users, including nodes for model I/O (STT, LLM, TTS), data engineering (prompt building, chunking), flow logic (keyword matching, safety), and external tool calls (MCP integrations).
- View end-to-end traces and logs automatically: Instead of reconstructing the execution path manually, developers simply go to Inworld Portal to view the end-to-end trace and logs. Every node execution is automatically instrumented with OpenTelemetry spans capturing the node, inputs, outputs, duration, and success/failure.
- Write modular, easy-to-understand code: Developers define each node’s inputs, outputs, and dependencies in a graph, making the execution path explicit and visible — you can see exactly which nodes connect to which others, making onboarding new team members easy. They can contribute to a single node on day one, then gradually understand the broader graph structure.
- Read Wishroll Status Case Study: Went from prototype to production in 19 days with a 20x cost reduction
Problem 3: Slow iteration speed
Before: Customization incurred technical debt
- Bespoke customization: As our customer base grew, so did the need for customization, so the code became brittle and hard to reason about.
- If/else hell: Different clients required slightly different logic, tools, or model choices. In our traditional codebase, this led to a labyrinth of if/else code blocks and feature flags scattered throughout the logic.
With Runtime: Fast user experience iterations
- One-line change for models and prompts: Want to swap an LLM provider or adjust a model parameter? That's a simple configuration change. A/B test variations and deploy customizations without touching production code.
- A/B testing at scale: We define agent behavior declaratively in JSON or through a fluent GraphBuilder API. Different clients get different graph configurations—not different code paths.
Why We're Sharing Inworld Runtime with You
We built Inworld Runtime to solve our own massive challenges in creating production-grade, scalable realtime conversational AI. But in doing so, we created a solution for a problem every AI developer faces: managing the inherent complexity of the "reason—act" agent cycle.
We believe the future of AI is not just about more powerful models, but better orchestration. It's about giving developers the architectural foundation they need to build robust, maintainable, and observable realtime conversational AI, without reinventing the wheel.
If you're tired of wrestling with tangled logic and want to focus on creating, we invite you to build your next experience on Inworld Runtime. Let us handle the complexity of orchestration, so you can focus on bringing your ideas to life.
Get Started with Inworld Runtime
Inworld Runtime is the best way to build and optimize realtime conversational AI and voice agents.
You can build realtime conversational AI that is fast, easy to debug, and easy to optimize via A/B experiments.
Get started now with Inworld
CLI: