Courtesy of Pixomondo, alter ego, William F. White Int’l, Virtual Production Academy, Caledon FC

Pixomondo gives viewers a kick with new virtual production-based soccer commercial

Back in the golden age of Hollywood, if a director needed a crowd for a scene, they’d have to bring in real actors. Ben Hur’s famous chariot race saw 8,000 screaming extras fill the stadium, for example. By the time Gladiator was released, it was possible to use CG to populate the Colosseum. The former approach is an expensive, logistical nightmare; the latter can involve hours of laborious digital artist work. 

When you’re shooting something like a commercial, budgets are lower than for Ben Hur. Renting a stadium and bringing in hundreds or thousands of extras is off the table. Spending hours in post-production is an option, but not a particularly encouraging one.

Fortunately, there’s now a way to create CG crowds that are indistinguishable from the real thing at a fraction of the time and expense. 

On a recent commercial for the Caledon Football Club, Pixomondo (PXO), alter ego, William F. White International (WFW), and Virtual Production Academy (VPA) showed how, cutting costs—not quality—with an LED volume and virtual production workflows.
 

Slow motion on an LED stage

If you need a screaming horde, David Whiteson is your man. The Head of VFX at alter ego has been working in the industry for the past 26 years as a compositor. He also puts on his director’s hat when required, co-directing the Caledon Football Club commercial.  

One of the biggest technical challenges Whiteson’s team had to overcome on the project related to frame rates. Shooting at high speeds can cause a flickering effect on the LED screens. 

Why did they need to shoot at such a high frame rate? Because time slows down as the commercial’s lead character leaps to execute a bicycle kick and scores the winning goal. Slow-motion footage is shot at a high frame rate, and then played back at lower frame rates. 
 
Courtesy of Pixomondo, alter ego, William F. White Int’l, Virtual Production Academy, Caledon FC

“As far as I am aware, slow motion has never been achieved on an LED stage before this commercial,” says Whiteson. “When running higher frame rates, like 48fps on LED walls, the likelihood of seeing scanlines in-camera becomes more prevalent. However, this is also contingent on your LED panels, cameras, and the movement of these cameras. To achieve the best results, we tested three different cameras at various different frame rates while the wall was running at either 24fps or 48fps.”

In the end, the team decided to use the Alexa Mini because it can do 200fps and produces the least amount of flicker. “No other camera other than Phantom could do that for us without cropping in on the sensor,” says Whiteson. “Even at 200fps, we still needed to slow it down to 800fps.”

In addition to the flicker, the team also discovered that the crowd simulation would need to run slower than the original 24fps. Once they shot their first test, it became clear there was not enough information in the crowds movement to withstand the 200fps camera speed, so they prebuilt different speed variations of CG crowd. At any given moment, they could instantly activate one of the animations, depending on the frame rate they were shooting at.

“That's a huge benefit of using Unreal Engine—to load those different assets at the push of a button,” says Whiteson.
 

Digital 3D crowds for film and TV

As a compositor, Whiteson is often asked to duplicate or generate 3D crowds to fill a stadium—a task that he says is both time-consuming and challenging. “It’s always tedious work,” says Whiteson. “Tracking, keying, or rotoscoping, cloning shot elements, blending in atmosphere—it’s something we are asked to create maybe four to five times a year.”
 
The team at alter ego wanted to provide a cost-efficient solution to clients that would give them the freedom and creativity to tell their story—without huge post implications that tend to follow crowds.

That’s where PXO came in. Earlier this year, the award-winning international visual effects and virtual production company completed over 700 VFX shots for the HBO series Winning Time: The Rise of The Lakers Dynasty

As you can imagine, 90% of the work consisted of CG stadiums and crowd simulations. The PXO team realized there was a need for a faster and more efficient way of doing 3D crowds. In the film and TV industry, ‘faster and more efficient’ often means using game engine technology, so the team developed a flexible crowd tool in Unreal Engine that could not only perform in real time, but be used on LED volumes.

When the opportunity to work on the Caledon Football Club commercial came up, alter ego and PXO knew they had to put this new tool to the test and really push it to its limits.

“PXO and alter ego have collaborated on several commercials,” says Pixomondo Chief Innovation and Chief Creative Officer Mahmoud Rahnama. “After PXO wrapped the first season of HBO’s Winning Time, I told David about the challenges we had creating the VFX crowd sequences using traditional methods. He told me about his challenges attempting to shoot high frame rate shots on an LED wall. That’s when we decided to score two goals with one ball!” 
 

The resulting commercial takes place in a packed soccer stadium—a perfect test for crowd work. We follow the action as a young boy weaves past a number of opposition players before scoring a goal in front of hordes of screaming fans. Those fans are CG—but you’d never know it. The stadium and 18,000 real-time crowd agents are all generated in Unreal Engine. 

Real-life crowds are incredibly complex. Like a swarm, they can move in unison—cheering a goal or booing a bad refereeing decision—while at the same time, each individual has their own idiosyncratic movements, like turning to chat to friend or diving into a bag of chips.

The system PXO has developed enables them to capture the essence of this complexity. The team can change characters, LODs, jerseys, behaviors, density, and make the crowd do a wave, all in real time. And that’s just the start. They are working on making the tool more interactive so the crowd can follow the real performers within the volume; reacting to scores and even following the ball.
Courtesy of Pixomondo, alter ego, William F. White Int’l, Virtual Production Academy, Caledon FC
The PXO’s crowd system works by leveraging efficient static mesh instancing in the engine. All animation was done in Vertex Shader using bone animation textures. The system consists of two parts—a Houdini toolset for model and animation processing along with bone animation texture baking, and an Unreal Engine plugin, written in C++, to generate and control the crowd instances.

For those moments when the crowd needs to move as one, the system supports triggering different crowd emotions. There are over one hundred different animation clips that can be used on any crowd person, which can be activated to affect the whole crowd or just one section of it. The system also supports smooth blending between animation states.

“The ability to control the crowd to perform exactly as needed on cue is huge,” says Whiteson. “We pre-animated seven different crowd performances to be used depending on the action of the play on the field. If we wanted the crowd cheering, we just loaded up that specific asset. Likewise, if we wanted them to be sitting and not too excited, that was also a pre-animated simulation. Much easier than trying to wrangle real life people to perform in unison and on cue.”
 
Courtesy of Pixomondo, alter ego, William F. White Int’l, Virtual Production Academy, Caledon FC

PXO Virtual Production Manager Jack Chadwick explains that the team leaned heavily on Unreal Engine’s Niagara VFX system throughout the development of the real-time CG crowd. “Unreal Engine was integral to our ability to build a real-time full-CG dynamic crowd,” he says. “We were able to leverage Niagara to build a crowd that felt authentic without being repetitive. One of our biggest challenges was capturing the minor nuances of a real-life crowd within a simulated system being fed by a defined number of character animations.”

PXO motion captured all the animations, while the 3D models of people in the crowd were created from 3D scans of real people. “We spent a lot of time in our Xsens motion capture suits trying to find enough diversity in our animations to help bring our crowd to life,” explains Chadwick. “We were able to iterate and adapt quickly by leveraging the real-time render capabilities of Unreal Engine.”

“Everyone on this project loves pushing the boundaries of real-time technology and discovering new ways to solve filmmaking problems. This commercial was living proof,” says Rahnama. “By teaming up, PXO, alter ego, WFW, and VPA not only cracked real-time virtual crowds, but came up with a better way to achieve slow-motion shots on an LED wall. This is what gets you up in the morning, and makes you want to go even bigger next time. With Unreal as our playground, I’m sure we can.”
 

LED stages for commercial shoots

The commercial was shot on Stage 6, one of PXO and WFW’s virtual production stages in Toronto. This 2,000 sq. ft. semi-circular volume features 720 LED panels and 175 ceiling panels. Filmmakers are able to utilize a combination of live action and cutting-edge digitally created environments rendered in real time on its LED walls and ceilings.
Courtesy of Pixomondo, alter ego, William F. White Int’l, Virtual Production Academy, Caledon FC
Prior to LED walls, a project like the commercial for Caledon Football Club would have required alter ego to bring between 50 to 100 people into a real-life stadium, move them around from section to section, and then put it all together in post.

Or they would have to generate the crowd in CG, track the footage, rotoscope any foreground assets, and try to integrate it all together by matching lighting and atmosphere. “If creating a CG crowd needs to happen either way, why not spend the time creating it before the shoot and use Unreal Engine to animate in real time on the LED wall?” says Whiteson.

This way, the team is free to move its camera however it chooses because everything captured in camera is final pixels—what’s more, the blending between the real world and the environment on the wall is something that can not be matched in post. 
For the soccer commercial, alter ego inserted real footage of smoke into the CG stadium environment while also filling the real stage with smoke, so that both elements felt like they came from one source. It was the same story with the lighting. 
Courtesy of Pixomondo, alter ego, William F. White Int’l, Virtual Production Academy, Caledon FC
“The stadium lights in the Unreal environment reacted in the lens just as real stadium lights would,” says Whiteson. “The bokeh and light wrap around our athletes that we captured in camera was as true to life as if we were in a real stadium. Again, this is something that’s time consuming and difficult to replicate in post production.”
Courtesy of Pixomondo, alter ego, William F. White Int’l, Virtual Production Academy, Caledon FC
The scale of the crowd made it impractical to use traditional filmmaking methods on the project. “We achieved something that would have been impossible to capture in a real location,” says Whiteson. “We have more than 18,000 digital fans cheering in our stadium. There is no chance of hiring that many people in a real stadium.”

The team used Unreal Engine to create an entire 360° environment and move it on the LED wall to suit their framing—rather than moving the camera. “In a real location, we would have had a camera moving all over the field in order to capture the feeling of a real game being played,” explains Whiteson. “With Unreal Engine, we were able to very easily rotate the environment to any area of the stadium within seconds. We would lay down our dolly track and move the world instead of taking the time to move the camera.”
 
Courtesy of Pixomondo, alter ego, William F. White Int’l, Virtual Production Academy, Caledon FC

Chadwick points out that working with a LED screen also saves huge amounts of time compared to traditional techniques. “One of the main advantages of shooting a project like this on a LED volume is the speed at which we were able to execute,” he says. “You can imagine that if you were shooting this in a real stadium, aside from being expensive, it would take very long to reset for each take. By leveraging the real-time render capabilities of Unreal, we were able to reset with just the click of a button.”
 

LED stages make impossible shoots possible 

Whiteson believes agencies and production companies have good reason to be excited about virtual production techniques and LED volumes. 

He explains that they are currently working on a project that would be impossible to complete without Unreal Engine and an LED wall. The story needs to take place in winter but the team are shooting in September. That puts four options on the table.

The first would be to fly to a far-off location where it is currently winter—incredibly expensive. 

Alternatively, they could shoot at a local ski resort at the end of the summer and then replace the entire background in post. This option is very costly and time consuming in post production. 

The third option would be to shoot in a studio on a blue screen. “With this option we are dealing with studio lighting and 3D post environments, and the integration never really looks perfect,” says Whiteson. “Additionally, the delivery schedule does not allow for much post time.”

The final option was to create the environment before the shoot in Unreal Engine and use an LED stage. “For us, the last option will produce the best results,” says Whiteson. “The LED wall is so powerful in its brightness that it really does do a lot of the lighting for you. 

“Plus, the integration between the talent and the background would look far superior to any composite. We can also use atmosphere and snow falling on set which could never be done in a blue screen environment. Have you ever tried to key falling snow? It’s a nightmare. For the budget and timeline, the LED wall was really the smartest option.”

    Get Unreal Engine today!

    Get the world’s most open and advanced creation tool.
    With every feature and full source code access included, Unreal Engine comes fully loaded out of the box.