Image courtesy of Mado XR

Unreal Engine 5 helps Aya Nakamura stun music fans in the Fortnite Soundwave Series

October 18, 2022
Twenty years ago, pop stars had a number of well-established channels through which to build an audience: record sales, radio plays, music videos, and live performances. Today, there’s an exciting new way for artists to connect with fans—virtual worlds. Pioneering pop stars are turning to the digital realm to delight existing followers and reach a whole new fanbase.

Aya Nakamura is one such innovator. The chart-topping French-Malian singer recently featured in the Fortnite Soundwave Series—a series of musical experiences featuring artists from around the world. 
 

Fortnite players were recently invited to join an experience built from the ground up using Fortnite’s Creative toolset. There, a groundbreaking music experience saw an artist from France perform in Fortnite for the first time. 

Aya and her record label Rec. 118 / Warner Music France collaborated with XR studio MADO XR to create a digital replica of the star with help from MetaHuman Creator, while Unreal Engine 5 was used to build the virtual environments for the music videos that were shown in the experience.

MetaHuman for realistic avatars

“When I heard about this project, I immediately realized what a breakthrough it would be,” says Louis de Castro, Founder, CEO, and Creative Director of MADO XR.

For De Castro, Aya Nakamura’s Soundwave Series performance perfectly encapsulates how the digital world is becoming a canvas for music artists. “I picture it as an enormous playground that boosts your imagination and also allows you to look at your performance from a different angle,” he explains. “Being part of a huge project that demonstrates all the power of the metaverse is like being a pioneer. What can be more exciting?”

The Aya Nakamura virtual experience represents a milestone both for Aya and for French music—a point not lost on Aya herself. “The collaboration is an opportunity to perform my biggest show ever to fans old and new and from all around the world,” she said before the experience. “I can’t wait to share my music with you all and be the first French artist to collaborate with Fortnite!”

Alain Veille, President of Warner Music France, explains that music experiences hosted in virtual worlds like Fortnite provide a unique opportunity for artists to meet fans where they are already spending their time. “Bringing artists to a game like Fortnite, which players love, is a way to get them to connect more directly with their fans who share a common interest,” he says. “ ‘I had no idea they were a Fortnite fan’ is a common positive refrain we hear from fans about their favorite artists.”

As well as providing new and exciting ways to connect with fans, virtual experiences can offer a far more practical way for busy music artists to reach millions of people at once. “We often have more flexibility to work around the artist’s schedule to get things done, compared to a live tour passing through many cities and venues,” says Veille.
Image courtesy of Warner Music Group
Aya’s team were able to take this into account in the project timeline, and it even inspired the approach of recreating the artist as an avatar using MetaHuman Creator as part of the process. “The production work was more flexible than the in-person requirements of a live concert or tour,” explains Veille. 

Virtual worlds are a new frontier in music. Everyone from labels and artists to fans are on a journey together to find out what these digital spaces mean for the artist-fan relationship. 

As Warner Music Group’s Chief Digital Officer Oana Ruxandra explains, that means learning hand-in-hand with artists, their teams, their fans, and the platforms. “This project with Aya saw us experiment with new tools to enable her to build her vision more holistically,” she says.

“For example, this was the first time we used the MetaHuman tool within an Unreal Engine 5 environment. The fidelity of animation, our ability to capture Aya’s real movements within this virtual space was astounding to see and had us all incredibly excited and inspired.”
 

Unreal Engine 5 for virtual worlds 

MADO XR developed the visual concept, script, and avatar of the artist in collaboration with talented veterans of the music and digital industries.

The team worked with director duo The Quiffs to hone the script for the music video. Paris-based 3D scanning studio Scan Engine were tasked with scanning Aya to get the digital data that creative digital studio Shape Shifters Creative would use to craft her avatar. The virtual environments for the music videos that would be broadcast on screens in the experience were developed in collaboration with a team of French artists from Komet Xperience
Image courtesy of Warner Music Group
The first step of the process was the scanning. The team took more than 150 pictures of Aya’s face and body to create a high-fidelity scan that was transferred into MetaHuman Creator, a process that is now accessible to anyone through the Mesh to MetaHuman feature of the new MetaHuman Plugin for Unreal Engine. “The most challenging part of the process was the details that make her human—adjusting the color of the lips with the skin tone and face shadows, so we could replicate that in MetaHuman Creator,” says De Castro. “The next step was to reassemble the facial rig to make her emotions appear. After that, we proceeded to animation through motion capture.”
 
Courtesy of Warner Music Group

The process of reassembling the facial rig described above can now be automated. De Castro believes MetaHuman Creator has changed the game when it comes to creating digital humans. “The best part is that it allows any type of creator, whether it is a skilled 3D artist or a beginner, to get involved in avatar creation,” he says. “It’s user-friendly and intuitive, and the guidance provided is enough to shape a decent avatar even for those who have no previous acquaintance with technology.”
Courtesy of MadoXR and Waner Music Group
As an early adopter, De Castro has seen how much digital human technology has evolved from the traditional pipeline in which you had to redo the facial rigging, animation, and bones to achieve a facial expression that would mimic a real human. “In that sense, MetaHuman Creator is truly a one-shot tool,” he says. 

The best litmus test of the team’s success is the reaction of the artist herself. “I was really surprised when I discovered my avatar,” says Aya. “It looks so real—especially the movements and gestures. I instantly recognized myself, it’s really impressive!”
Aya’s performance was broadcast on screens throughout a virtual environment built inside of Fortnite. Developed by Atlas Creative Studios, players were able to explore this virtual world and experience Aya’s music at the same time. 
The team at Atlas worked hard to recreate the look and feel of the CG visuals in the music video for the virtual world that the players would run through. With the digital Aya and UE5-built environments of the music video resulting in a more photorealistic look that you would ordinarily expect to see in Fortnite,  the team had to dive into Fortnite’s Creative toolset to achieve a good match. 
Courtesy of Mado XR
“A lot of things we had to make don't necessarily look like a typical Fortnite map, because we had to tweak a lot of lighting and different things to match the video,” explains Michael Herriger, the studio’s co-founder and CEO.  “And that's what makes it really cool—you have this ‘you're in Fortnite, but out of Fortnite’ experience at the same time, which is exciting.”
 
Courtesy of MadoXR

The future of music experiences 

Ruxandra says that Warner Music believes the future will bring hyperreality metaverse experiences overlaid on ultra-thin headset devices. “These metaverse experiences will be persistent, synchronous, and live,” she says. “They will span the digital and physical worlds, offer unprecedented interoperability, be populated by content from a wide range of contributors, and have their own fully functioning economies.”

Within these metaverse experiences, Ruxandra continues, music will be a dominant form of interaction, communication, and consumption. “The possibilities are endless for our artists to engage with their fans in immersive, real-time, and customizable ways,” she explains. “We are incredibly excited to dig in here and now, at the outset of these future experiences, so that we can ensure our artists have the tools needed to become experts within these spaces.”

For De Castro, Fortnite is the natural destination for virtual music experiences like the ones Ruxandra describes. “Let's face it, no one puts on a better virtual show than the Fortnite team,” he says. “There ain't no party like a Fortnite party, and this has helped Epic Games set itself up as a forward-thinking host when it comes to making the best in-game concerts. 

“The platform recommended itself to us as a number one destination for those who want to bring together gamers, music artists, artists in general, and designers. With the metaverse buzzing, it's the best example of how digital reality works.”

    Get Unreal Engine today!

    Get the world’s most open and advanced creation tool.
    With every feature and full source code access included, Unreal Engine comes fully loaded out of the box.
    Join us online or in person at GDC 2024
    Event
    February 13

    Join us online or in person at GDC 2024

    Tune in to the State of Unreal for the latest announcements from Epic and our partners, dive deep with our tech talks, and get inspired and informed with our sessions and learning theater.
    Join us online or in person at GDC 2024
    Event

    Join us online or in person at GDC 2024

    Tune in to the State of Unreal for the latest announcements from Epic and our partners, dive deep with our tech talks, and get inspired and informed with our sessions and learning theater.
    February Marketplace sale: get 50% off over 2,000 products
    News
    February 15

    February Marketplace sale: get 50% off over 2,000 products

    Retro cafes, rustic interiors, low-poly towns, customizable NPC characters…February’s Marketplace sale has something for everyone. Act now to get this month’s selection of goodies at half price.
    February Marketplace sale: get 50% off over 2,000 products
    News

    February Marketplace sale: get 50% off over 2,000 products

    Retro cafes, rustic interiors, low-poly towns, customizable NPC characters…February’s Marketplace sale has something for everyone. Act now to get this month’s selection of goodies at half price.
    Featured free Unreal Marketplace content—February 2024
    News
    February 6

    Featured free Unreal Marketplace content—February 2024

    Fantasy interiors, stylized crystal mines, a trip back to the 90s, and more—February’s free 3D assets from the Unreal Engine Marketplace have landed!
    Featured free Unreal Marketplace content—February 2024
    News

    Featured free Unreal Marketplace content—February 2024

    Fantasy interiors, stylized crystal mines, a trip back to the 90s, and more—February’s free 3D assets from the Unreal Engine Marketplace have landed!