Sony Tests AI-Powered PlayStation Characters, Including Aloy Prototype

Sony tested AI-powered PlayStation characters, including Aloy, sparking debate on AI’s gaming role after a leaked video.

The Verge first reported on the video, which showcased an experimental AI version of Aloy responding to voice prompts with AI-generated speech and facial animations. Shortly after publication, the video was removed from YouTube following a copyright claim by Muso, a company linked to Sony Interactive Entertainment.

The video, narrated by Sony software engineering director Sharwin Raghoebardajal, demonstrates how the AI-powered Aloy can engage in conversations.

It utilizes OpenAI’s Whisper for speech-to-text, GPT-4 and Llama 3 for dialogue, Sony’s Emotional Voice Synthesis (EVS) for speech, and the company’s Mockingbird technology for facial animation. “This is just a glimpse of what is possible,” Raghoebardajal says in the video, as reported by The Verge.

Sony developed the prototype in collaboration with Horizon developer Guerrilla Games. While the demo was primarily tested on PC, Sony has reportedly experimented with running it on PS5 with minimal performance impact, says The Verge.

The technology was first showcased internally a year ago, with a more refined version presented behind closed doors at the Sony Technology Exchange Fair (STEF) in Tokyo last November, as reported by The Verge. The leaked footage has raised concerns within the gaming industry, particularly among voice actors and developers.

IGN reports how the AI-generated voice of Aloy does not belong to Ashly Burch, the character’s original actor, sparking debates on AI’s role in game development. Some viewers also noted that AI Aloy’s speech felt robotic, with stiff facial movements and lifeless eyes.

The broader gaming industry has been exploring similar AI implementations. Nvidia has been developing AI-driven non-playable characters (NPCs) under its “ACE” program, , allowing players to engage in real-time conversations with in-game characters.

AI-powered NPCs could transform open-world storytelling by making interactions more dynamic and immersive. Supporters argue that AI can personalize quests based on player choices, creating more meaningful experiences, as reported by Medium.

NPCs could remember past encounters, altering their dialogue accordingly. AI-driven characters may also adapt their tone and responses to match emotional situations, making conversations feel more realistic.

Additionally, AI can adjust difficulty levels and enemy behaviors in real-time for a balanced experience.

Microsoft, meanwhile, is partnering with Inworld AI to introduce generative AI features for Xbox developers, as noted by The Verge.

Additionally, Ubisoft’s NEO NPC, launched in March 2024 with Nvidia and Inworld AI, enables real-time, dynamic NPC interactions. Powered by Nvidia’s Audio2Face and Inworld’s language model, the AI syncs facial animations with speech, generating context-aware responses.

The rise of AI in game development is controversial. A recent Game Developers Conference (GDC) survey found that nearly half of respondents use generative AI in their workplaces, but concerns remain about its impact on jobs and creativity. Critics argue AI in gaming could weaken storytelling by replacing human-crafted narratives.

Medium reports that concerns include robotic dialogue, loss of emotional depth, and job displacement for writers and voice actors. While AI mimics patterns, it lacks true creativity, raising fears that games may lose the nuance and authenticity human storytellers provide.

IGN notes that companies like EA and Capcom have embraced AI for content generation, while others, such as Keywords Studios, have acknowledged AI’s current limitations, noting that it failed to replace human creativity in game development.

Sony has yet to announce plans to integrate AI-driven characters into PlayStation games. The timing of the leak—just ahead of GDC 2024—ensures AI in gaming will remain a key topic of discussion.