Taking Unreal Engine's latest in-camera VFX toolset for a spin

November 8, 2021
Recently, Epic Games and filmmakers’ collective Bullitt assembled a team to test out the latest in-camera VFX toolset, part of the extensive suite of virtual production tools in Unreal Engine 4.27, while it was still in Preview. To put each of the new tools through their paces, they created a short test piece to mimic a production workflow.

Shot entirely on NantStudios’ LED stage in Los Angeles, the production test capitalized on new workflows that enable dynamic lighting and background changes on set with minimal downtime, together with multi-camera and traveling vehicle shoots. It also harnessed the power of twin NVIDIA Quadro A6000 GPUs. The team was able to produce final pixels in just four days—a demonstration of the efficiency and creative freedom the toolset offers. 
 
The team also took advantage of new Level Snapshots that enable directors to easily match previous takes when returning to an earlier set or lighting configuration; sophisticated remote control UI building tools that offer creatives direct control of the Unreal Engine-powered elements of the shoot from the convenience of a tablet or laptop; and OpenColorIO color management that ensures artists creating Unreal Engine assets can see what the final captured output will look like.

In the first video in the series, the filmmakers talk about the huge benefits they witnessed when working with these latest updates to Unreal Engine’s in-camera VFX toolset. The entire team, including the director, producer, production designer, director of photography, and first assistant director, all agreed that what they experienced represented a quantum leap for on-set filmmaking.
 
The ability to find new shots on the stage is just incredible. In the same amount of time that it would take me to change a light in the physical world, I'm able to have new lighting on the wall. As a director, that's a game changer. 
- Ryan McNeely, Director, Visual Creatures
Recently, some intrepid filmmakers have been reaping the benefits of using in-camera VFX within LED volumes in productions such as The Mandalorian. With existing tools, however, digital set changes could require hours or even days of time for light baking, losing much of the spontaneity in the process. The new toolset in Unreal Engine 4.27 includes GPU Lightmass, which takes advantage of light baking on multiple GPUs to radically accelerate set changes, making them as much as an order of magnitude quicker. Instead of breaking for the day, teams can now just take a coffee break.
Ryan McNeely, who acted as director on the production test, explains why this is such a huge step forward. “As a director, time is your biggest enemy, and the thing about this, the speed at which we're able to move the amount of setups, that's honestly the dream,” he says. “The ability to find new shots on the stage is just incredible. In the same amount of time that it would take me to change a light in the physical world, I'm able to have new lighting on the wall. As a director, that's a game changer.”
For renowned filmmaker Anthony Russo, it’s the ability of in-camera VFX to bring everything together and enable the team to see the final result right on set. “One of the things that excites us most, of course, about this is the fact that we can do in-camera choreography, where all the elements of the frame are actually in concert with one another and organically working on one another to create a more visceral experience,” he says.

Director of Photography Matthew Jensen agrees. “It's so much easier now to be able to make the changes in real time, see their effects, and then make adjustments from there,” he says. “It allows for a happy accident. It allows for more spontaneity in the creative process.”
What’s more, in-camera VFX has the ability to remove the silos that can cause creative intent to get lost in translation. 

“It's not separated into components of pre-production, shoot, and post-production,” says Diane Castrup, Producer. “Usually you just have your physical set that you're standing on and the green screen is behind you and you just kind of give it over after you're done shooting and you hope that it all works out.”

Production Designer Quito Cooksey echoes these sentiments. “Once principal photography ends, post takes over,” he says. “And if you're not part of that, everything that you really envisioned and designed goes through a filter and doesn't always come out the other end in the way in which you envisioned. With this, you have the virtual art department with control of the concept designs all the way through to final pixel.”
Sean Harner, First Assistant Director, admits to being skeptical when he first arrived on set. He’s now a convert. “I am totally blown away with how you can move things around, how we can change sun direction, how I can physically change sets,” he says. “We’re taskmasters, and it's all about time. If I lose an hour on a commercial or a movie, I can't get that time back and that could be $100,000 or that could be $500,000.”
It was photoreal and it was ready to reel. That means that you could go from shooting it to projection in the same day. That's technology at its finest. 
- Sean Harner, First Assistant Director, 2H Productions
Another huge advantage of the new toolset is the ability to quickly and easily recall a set that you’d shot earlier—perfect for those additional pickups, as Harner discovered when working on the project. “And just in between the sets is 20, 30 minutes’ setup,” he says. “It was photoreal and it was ready to reel. That means that you could go from shooting it to projection in the same day. That's technology at its finest.”

The last word goes to Joe Russo. “It feels like we're moving from filmmaking 1.0, jumping to filmmaking 5.0—and I don't see any other path forward that is nearly as exciting, compelling, or freeing for artists as this path,” he says.
 
At Epic, we have an extensive virtual production team who are dedicated to finding better ways for filmmakers to work. Here, we chat to some of them about the Unreal Engine 4.27 features used in this production test.
 
Unreal Engine 4.27 is available now; it's free to use for creating linear content like films and television shows. The exciting new workflows and tools it offers are influenced by input from leading filmmakers who are pushing the limits of virtual production today. As the Bullitt team discovered, the toolset represents an evolutionary leap forward in quality, efficiency, and ease of use—delivering whole new levels of creative freedom. 

To explore the toolset for yourself, download the full sample project from this ICVFX production test and take a look at the full tutorial series that goes along with it.

    Ready to try Unreal Engine 4.27’s ICVFX toolset?

    Download the full sample project from the production test and check out the full tutorial series that accompanies it today.
    Join us online or in person at GDC 2024
    Event
    February 13

    Join us online or in person at GDC 2024

    Tune in to the State of Unreal for the latest announcements from Epic and our partners, dive deep with our tech talks, and get inspired and informed with our sessions and learning theater.
    Join us online or in person at GDC 2024
    Event

    Join us online or in person at GDC 2024

    Tune in to the State of Unreal for the latest announcements from Epic and our partners, dive deep with our tech talks, and get inspired and informed with our sessions and learning theater.
    February Marketplace sale: get 50% off over 2,000 products
    News
    February 15

    February Marketplace sale: get 50% off over 2,000 products

    Retro cafes, rustic interiors, low-poly towns, customizable NPC characters…February’s Marketplace sale has something for everyone. Act now to get this month’s selection of goodies at half price.
    February Marketplace sale: get 50% off over 2,000 products
    News

    February Marketplace sale: get 50% off over 2,000 products

    Retro cafes, rustic interiors, low-poly towns, customizable NPC characters…February’s Marketplace sale has something for everyone. Act now to get this month’s selection of goodies at half price.
    Featured free Unreal Marketplace content—February 2024
    News
    February 6

    Featured free Unreal Marketplace content—February 2024

    Fantasy interiors, stylized crystal mines, a trip back to the 90s, and more—February’s free 3D assets from the Unreal Engine Marketplace have landed!
    Featured free Unreal Marketplace content—February 2024
    News

    Featured free Unreal Marketplace content—February 2024

    Fantasy interiors, stylized crystal mines, a trip back to the 90s, and more—February’s free 3D assets from the Unreal Engine Marketplace have landed!