Spatial computing meets AI in our Apple Vision Pro Unity module

Apple devices have transformed the way we live, work, and play. Now, the Apple Vision Pro is now opening up possibilities for us to interact with digital content in our physical world like never before. As Apple puts it, the Vision Pro introduces, “a new dimension for entertainment,” that when married with AI technologies like the Inworld Engine, will enhance user engagement and narrative depth.

Inworld’s Apple Vision Pro Unity module lets developers and creators craft more immersive and dynamic environments by adding Inworld-powered AI agents and AI NPCs to their mixed reality applications. The integration uses Unity Polyspatial, a popular framework supported by Unity and Apple to build Apple Vision Pro games and experiences.

Apple Vision Pro + AI = Deeper immersion

With features like high-resolution cameras, image anchoring, and sophisticated eye, head, and hand tracking, the Apple Vision Pro Unity Module provides Inworld with a rich stream of data about the user's environment and behavior. This allows our AI to perceive and understand the context of the user's actions in the mixed reality space, enabling more nuanced and dynamic responses. 

With the Apple Vision Pro, AI characters can see and sense the world around them, adding important contextual information to character cognition. Developers can leverage these multimodal data streams (speech, text, player gestures, finger tracking, and gaze) to build intuitive and magical Apple Vision Pro AI character interactions.

For example, Inworld characters can respond by saying hello when the player waves. Or making quippy comments when players glance at virtual objects in the environment. Imagine if an NPC could remark on the new sword you’ve acquired or recognize that the armor you’re wearing is from a warring faction. In an Apple Vision Pro game powered with Inworld, they now can.  

See the Inworld Apple Vision Pro Unity module in action

Add depth and engagement to Apple Vision Pro games and mixed reality experiences

Whether you’re building Apple Vision Pro games and experiences or building for WebAR, Inworld’s AI agents, characters, or brand ambassadors can add an additional layer of engagement and realism to mixed reality experiences. Here are some ideas for getting started:

• Interactivity and engagement: Respond to user actions and gestures changes in real-time, creating moments of delight and surprise like in Liquid City’s Wisp World 
• Narrative Depth: Add depth and complexity to your story which characters that react to users in ways that advance the plot, adding a level of interactivity to the storytelling.
• Personalization: Learn from user behavior and preferences, allowing them to adapt and provide personalized experiences like in Ubisoft’s NEO NPCs prototype
• Social interaction: Simulate social interaction, providing company, conversation, and even emotional responses
• Educational and training applications: Act as tutors or guides, providing instruction, feedback, and assistance tailored to the user's needs like in Niantic’s Wol experience or with LG’s Kidstopia

An Apple Vision Pro x Inworld AI application from Liquid City called Wisp World

Access the Inworld Apple Vision Pro Unity Integration Module via GitHub repository for the code and TestFlight for a playable demo of our Apple Vision Pro game demo.