Cost per Completed Interaction: The Consumer AI Inference Scorecard

TL;DR: Cost per token does not describe the cost of a consumer AI product. The useful unit is cost per accepted completed interaction: every input, cached token, output, retry, failed request, moderation check, memory update, tool call, and voice step required to deliver one outcome the user accepts.

A model price looks simple because it has one unit. A consumer interaction does not. A single turn may include routing, safety, retrieval, memory, a main response, tool calls, and speech. The same user may return dozens of times, carrying more context into every session.

Why does cost per token mislead?

Token prices compare one billed unit, but providers and models produce different output lengths, cache behavior, latency, quality, and retry rates. A lower-priced model can cost more when it needs extra attempts or produces fewer acceptable results. A higher-priced model can be cheaper when it completes the task once.

cost per accepted completion = total inference spend / accepted completed outputs

For an application, expand the numerator to include every model call in the interaction. Include moderation, retrieval generation, memory, ranking, tool selection, the main answer, speech, retries, and failed calls. Then divide by the number of interactions that met the product’s quality and latency requirements.

This unit can be connected to revenue, retention, session length, or another product outcome. Cost becomes an operating metric rather than an API invoice.

How do caching and routing change cost?

Long conversational prompts often repeat system instructions, memories, policies, and history. Prompt or prefix caching can bill repeated input at a lower rate and avoid duplicate prefill work. The blended saving depends on the percentage of input tokens that hit the cache.

blended input = uncached input × standard rate + cached input × cache-read rate

Routing can lower cost by assigning simple tasks to smaller models and reserving expensive models for difficult requests. It can also increase cost when retries, extra hops, or poor timeout settings duplicate work. Record the model selected, attempt chain, routing reason, and billed tokens for every request.

The live Inworld model catalog lists input, output, cache-read, and context values by model. Because those rates change, every cost table should include the model ID and verification date.

Which variables belong in a forecast?

Variable	Example unit	Reason
Daily active users	Users/day	Scales total interactions
Sessions per user	Sessions/user/day	Captures repeat behavior
Interactions per session	Turns/session	Captures conversation depth
Calls per interaction	Model calls/turn	Includes support tasks
Input and output	Tokens/call	Drives usage
Cache reuse	% input tokens	Changes work and price
Retry and failure	% requests	Adds hidden spend
Acceptance rate	% outputs accepted	Adjusts for quality

Run low, expected, and high scenarios. Consumer traffic is volatile, so a single forecast can conceal the risk of viral growth or seasonal bursts.

When does reserved capacity cost less?

Serverless or pay-as-you-go inference fits uncertain and variable traffic because the team pays only for usage. Reserved or dedicated capacity can lower effective unit cost and improve consistency when utilization is high enough. The break-even point depends on commitment price, usable capacity, traffic shape, and overage behavior.

Do not calculate utilization from average traffic alone. A consumer product may have low daily average utilization and sharp peaks that require more capacity. Include the cost of idle reserved hardware, burst capacity, failover, and engineering operations.

Inworld, Together AI, Fireworks AI, Baseten, and major clouds expose different combinations of shared, provisioned, dedicated, or reserved inference. Compare the actual product, region, SLA, and contract rather than treating the labels as equivalent.

How should quality-adjusted cost be measured?

Quality-adjusted cost divides spend by outputs that meet an application-specific rubric. The rubric can combine factuality, safety, style, tool correctness, latency, and task completion. It should be scored consistently across models and versions.

Suppose Model A costs half as much per token but only 70% of outputs pass, while Model B passes 95%. The effective cost gap narrows once rejected outputs and retries are included. The same logic applies to latency: an answer that arrives after the user abandons the turn has no useful completion value.

Run evaluations before and after quantization, caching, speculative decoding, or routing changes. Infrastructure optimization is not successful when it saves compute by reducing the value of the answer.

What do Inworld customer examples show?

Status by Wishroll reached one million users in 19 days, turning rapid adoption into an inference-cost problem. Inworld reports more than 95% lower AI costs while maintaining or improving content quality. The result combined targeted prompts, application-specific evaluators, production load testing, model work, routing, and lower serving cost.

Latitude’s Voyage product reportedly makes roughly 20 model calls in one interactive-fiction turn. The case illustrates why per-token price alone cannot explain product economics; the architecture determines how many calls occur and which model handles each one.

Both are Inworld-published customer examples. Metrics and causal claims require customer approval and should not be presented as independent validation.

Related Guides

Key Takeaways

Cost per token omits retries, failures, support calls, cache behavior, quality, and interaction depth.
Cost per accepted completed interaction connects provider spend to a user-visible outcome.
Cache discounts matter only when the workload produces material token reuse.
Reserved capacity wins when utilization and consistency benefits exceed commitment and idle-capacity costs.
Every infrastructure optimization should pass the same quality and product-outcome evaluations as its baseline.

Frequently Asked Questions

What is inference cost?

Inference cost is the expense of running a trained model to process input and generate output. APIs usually meter tokens, characters, audio time, images, video seconds, or compute. Application cost also includes retries, routing, support calls, idle capacity, and engineering operations.

How can a consumer AI app lower inference cost?

Start with workload measurement. Shorten or retrieve context, increase safe cache reuse, route tasks to appropriately sized models, reduce failed calls, batch asynchronous work, and compare reserved capacity at realistic utilization. Re-run quality evaluations after every change so savings do not reduce accepted completions.

Is a cached token always cheaper?

No. Many providers publish lower cache-read rates, but policies differ by model and caching method. The application also needs a matching prefix within the cache lifetime. Measure cached input tokens and the resulting bill rather than assuming a provider discount applies to the entire prompt.