October 4, 2019
Real-time talking animals with Demodern
In the Eye to Eye VR experience, the participant converses with an engaging and lifelike CG rabbit at eye level. The animal’s facial expressions and dialogue are driven in real time by an actor, giving the opportunity for real conversation with the participant.
Capturing expressions and emotion
The impact of the Eye to Eye VR experience comes largely from the way it succeeds in imbuing a CG animal with human-like qualities. Advanced facial tracking, courtesy of FaceWare Live, and body tracking via Noitom’s Perception Neuron suit, transfer every nuance of expression from the actor to the rabbit. This, combined with an extraordinary level of detail in the real-time rendering, delivers a uniquely emotional VR experience to the participant.
The requirement for the highest-quality rendering, coupled with the frame rate needed for virtual reality, is one of the reasons the team at Demodern turned to Unreal Engine.
“Unreal allows us to achieve the highest degree of visual fidelity while keeping silky-smooth 90 fps,” says Technical Director Florian Wögerer.
In order to achieve the highest performance possible, the team uses two computers to run the experience: one for the motion-tracking tasks, and one for the actual rendering, which features two NVIDIA GTX 1080 Ti cards and uses AFR (Alternate Frame Rendering) as a rendering method. Tracking data is transmitted to the rendering machine over a cross-link LAN cable with an SLI (Scalable Link Interface) bridge.
To model and texture the rabbit character, the team used a slew of packages including Maya, ZBrush, 3ds Max, TopoGun, Mari, Substance Painter, Photoshop, and Knald. The fur was groomed using Peregrine Labs’ Yeti, and prepared for real-time rendering in Unreal Engine using NVIDIA HairWorks. The character took about 500 hours to complete.
To incorporate HairWorks and another NVIDIA API, VRWorks, the team created a custom build of Unreal Engine, taking advantage of free access to the source code.
“Having the entire code available is definitely one of the major advantages of Unreal Engine,” says Christopher Baumbach, Senior Creative Engineer at Demodern. “It gives us the opportunity to adjust and optimize certain parts of the engine to achieve the desired quality and performance.”
More than just a model
For the environments, the team conceptualized, modeled, and developed five different worlds in Unreal Engine. To achieve the looks they wanted, the team at Demodern used Unreal Engine’s Material Editor.“Using the Material Editor and its flexible way of creating simple as well as complex materials was one of the key elements to create the look we were aiming for," says Daniel Harrison, Lead 3D Artist.
A key component of the experience is the sound, provided by Hamburg-based sound design studio German Wahnsinn. The team used FMOD as a sound engine, and edited the sound with Nuendo. The actor’s voice runs through an Apollo Twin audio interface, which has a built-in effect stack that allows noise-gating, compression, and so on. Pre-recorded audio is also used.
To achieve the performance and quality that the project required to be successful, the team worked through many technical challenges, resulting in a custom hardware and engine setup that went through several iterations before reaching its final state. The ability to build incremental changes in Unreal Engine was something that helped with this process.
“Being able to iterate quickly, without the need to create full builds every time we made some changes, gave us enough time to explore different approaches to tackling the technical challenges we were facing,” says Baumbach.
The experience was launched at re:publica '18 in Berlin and has been presented at several other festivals and trade shows. It is built into a truck that can quickly go to any location with minimal additional effort.
Taking a deep dive into the Elbe river
Since completing the Eye to Eye project, Demodern has also used Unreal Engine for one major part of the Hamburg-based permanent installation Discovery Dock. The real-time experience provides visitors with a convincing look into the depths of the Elbe, Hamburg’s famous river, and shows its most common animals.
Working with the Dumont Media Group, the team took the project from initial research to final exhibition in less than a year. They transformed an empty space next to the Elbphilharmonie concert hall in Hamburg into an immersive, interactive experience that employs a unique choreography of mixed reality, projection mapping, and real-time data. It takes visitors from Hamburg and all over the world to places in the port they’d never have access to—in a way they won’t easily forget.
Want to create unforgettable experiences of your own? Download Unreal Engine today.