Image courtesy of Framestore

How to teach virtual production

October 26, 2021
Today’s film students will create the Oscar-winning dramas, summer blockbusters, and binge-worthy series of the future. What if they could shoot them on a stage that could pause a sunrise, bring dragons to life, or move actors to dozens of locations in the moment?

That’s what will happen if they use virtual production (VP). From The Mandalorian to Westworld, VP is changing the way content is created. With a global market value that’s projected to reach $4.73 billion by 2028, it’s now capturing the attention of schools everywhere. Sheridan’s SIRT Centre just launched its own VP Innovation Hub (VPIH), while Escape Studios made their own VP short as a case study for an upcoming training. SCAD even invested in a next-generation XR stage as part of a new 10.9 acre film studio complex.

Everywhere you look, you see momentum. But you also see questions. Lots of them from educators trying to get a handle on how to build the most effective curriculums.

We can help!

Below you’ll find advice from educators and studios who have been putting VP through its paces—from a practical and educational standpoint—so you can start designing a program that you not only like, but will work.

How to Get Started 

Know the options
There are many roads to successfully teaching virtual production. A great place to start, however, is by looking at what you know. In most cases, a VP program can be slotted into what the school already teaches. 

“If you have been teaching games, then take a game level, look at how it differs from what is needed for VP, and work from there,” explains Mark Flanagan, Education Partner Manager at Epic Games. “If you’ve been teaching on-set lighting, start by exploring how to apply those same theories to lighting in Unreal Engine.” 

The type of VP you choose can also make a huge difference to your curriculum. There are various different types of VP, including: 
  • Visualization: using imagery to convey the creative intent of a sequence. Visualization can take the form of previs, virtual scouting, stuntvis, and more.
  • Motion capture: recording the movements of objects or actors, then using that data to animate digital models (think Gollum in Lord of the Rings).
  • Hybrid virtual production: using camera tracking to composite green screen cinematography with CG elements.
  • LED virtual production: replacing green screens with LED walls that can display CG elements and images from real-time engines. This means everyone on set can see and capture final-pixel imagery, completely in camera. Popularized by The Mandalorian, the use of in-camera VFX (ICVFX) will only continue to grow. 

The right VP for you can depend on many factors, from your budget to the physical space at your school. According to Manon Hartzuiker, Virtual Production Producer at Framestore, visualization is a good place to start. “It sits at the top of the production process and leads into other areas of the workflow,” she explains.
Image courtesy of Oger Sepol
“Focusing on performance capture and real-time visualization will also give students a strong foundation in the production experience,” agrees Scott Meadows, Head of Visualization and Virtual Production at Digital Domain.

For those who want to take things one step further, using a hybrid green screen solution is generally the most effective way to keep costs under control, while LED volumes are best for demoing high-end productions.
Get the tech
Practice makes perfect. That’s why students also need access to all the tools used on a real virtual production set. As well as the inclusion of Unreal Engine, setups can consist of:
  • VR headsets like HTC Vive or Oculus
  • Camera tracking systems like Ncam or Mo-Sys StarTracker
  • Motion capture systems like Xsens or OptiTrack
  • LED panels 
  • Green screens
  • Witness cameras

Once you’ve decided what type of VP you will teach, it’s important to make an inventory of the resources you need, based on the budget and available space at your disposal. 
Image courtesy of Framestore
Finding and training educators
The last piece of the puzzle is the most important to get right. Whatever type of VP you choose and tech you buy, it’s crucial to hire new staff with VP experience from the industry, or train your current team to a high level.
Image courtesy of Animism Studios
Luckily, there are a number of resources available to help, including free Unreal Engine content packs and Quixel Megascans environments that anyone can download to test real-time workflows for themselves.
Image courtesy of Dimension Studios and DNEG
For Spencer Idenouye, Virtual Production Lead at SIRT, the Unreal Fellowship program was the perfect way to train. “It was a fantastic opportunity,” he remembers. “In five weeks, I was able to construct a short film almost entirely using Unreal Engine and Unreal Marketplace assets. Participating in this sort of exercise, while documenting the process along the way, helped me to gain an in-depth understanding of the skills needed for virtual production.” 

Planning Your Curriculum

The top seven topics to teach VP students
With a technology that spans across as many applications as virtual production, it can be challenging to decide what to teach. Here are the most important topics to cover:

1. The fundamentals: It’s critical for anyone working with VP to understand the correlation between film production concepts and their game engine equivalents. That means learning about editing, directing, acting, and composition. “Learning VP does not mean you don’t have to know how to make movies,” adds Flanagan.

2. On-set lighting: Learning about lighting is also essential for VP students to create realistic backgrounds. “Matching the lighting of the physical foreground to the virtual background really brings everything together,” says Joerg Schodl, Cinematographer at SCAD.

3. Scheduling and production: Virtual production is making the industry re-evaluate pipelines that have been in place for decades. When VFX no longer happens in post, planning needs to be airtight. That’s why VP students need to learn everything they can about scheduling and production.

4. Virtual design: It’s not enough to teach students about onset filmmaking. They need to learn about CG too in order to create and prep content for real-time environments. “It’s important to teach students how 3D assets are built for both static meshes and skeletal-based characters through geometry, materials, rigging, blendshapes, and more,” explains Spencer Idenouye, Virtual Production Lead at SIRT.

5. Stage operations: In LED VP, the setup and engineering of the LED wall including color calibration, nDisplay setup, and more is critical. For students, learning the basics of how these work will give them a deeper understanding of what goes on behind the scenes.

6. Performance capture and camera tracking: If you want to animate 3D characters in real-time, or film actors on the moon, you need to learn all things mocap. Students should know how to ensure cameras are properly calibrated, which solutions they can use for streaming mocap data into Unreal Engine, and understand facial capture work.

7. Visual scripting: Familiarity with game logic and mechanics will help students develop or customize tools that will support their virtual production projects, and help them solve complex problems once they move into the industry.
Image courtesy of Neweb Labs
How to plan the most effective projects
Projects that emulate real-world scenarios in virtual production are the most effective for training students. “When a student can see their work on the big screen, that creates the strongest effect,” says Deputy Dean at Escape Studios, Saint John Walker. “The 1600 nits of a big LED wall make a lasting impression, even if you don't yet fully understand how everything works underneath. Film students, unlike their games colleagues, aren’t used to experiencing such immersion.”

Projects could range from conceiving, planning and executing a short film, to setting motion capture and real-time systems for shooting days. No matter what you choose, Digital Domain’s Meadows explains it’s important to focus on the process behind each task. “Students shouldn’t just be following instructions,” he adds. “They should be considering the steps for calibrating a stage, and identifying pain points to improve speed and stability.”
How to keep your curriculum current
When teaching an emerging technology like virtual production, keeping a curriculum current is a major challenge.
Image courtesy of Dimension Studios and DNEG
Educators need to consistently maintain close relationships with technology companies in order to keep up with hardware and software updates. “As new hardware is released, schools could even request early access and give feedback to developers as a way to stay on track,” Meadows explains. “Creating and participating in user groups and forums is also a great way to stay current.” 

According to Framestore’s Hartzuiker, it’s also important to keep in touch with VFX production teams. “Every production is different, and we're always finding new ways of making our workflows more efficient,” she says. “When schools reach out, we’re happy to share our findings.”

Setting Job Expectations

Following virtual production training, students will likely take on a variety of roles, such as:
  • Virtual Production Supervisor: Manages the on-stage component of virtual production technology to achieve fully integrated live action and CG hybrid shots. 
  • Technical Artists: Uses Unreal Engine to create virtual set extensions, optimize data for real-time playback, animate with Blueprints, and more.
  • Production Designer or VAD Artist: Takes advantage of virtual set scouting to finalize environments via VR & AR technologies. 
  • Previsualization Supervisor or Artist: Conceptualizes film projects, including establishing the creative direction of key sequences.
  • Mocap Animator: Captures, refines, and delivers realistic animation based on motion-captured data.

As well as the jobs above, a virtual production generalist will also have access to many other traditional filmmaking roles. “Everything in virtual production stems from a solid understanding of filmmaking, and conversely, any new knowledge or techniques obtained from learning virtual production can inform new processes for traditional filmmaking,” explains Brian Pohl, M&E Technical Program Manager at Epic Games. 

After VP training, for example, a student could decide to be a Director of Photography or gaffer after doing things virtually, before ever even touching a real light. Graduates could also expand into the role of the Digital Imaging Technician, or cinematography for the same reasons. 
Image courtesy of Last pixel Pty Ltd
“As a film teacher and filmmaker I see this as a tool for democratization,” says Schodl. “It can be expensive and time consuming to ensure students gain enough practical experience, but with real-time engines there are no limitations. Students can learn all the skills they need to work on a practical or a VP set—through a single curriculum.”

Learn More

Check out the full Epic Games Virtual Production Hub, and discover insider opinions, field guides, tips, inspirational stories, and more.

    Get more learning resources

    Sign up for the Education Newsletter to receive monthly updates and free, valuable teaching resources for integrating Unreal Engine into your curriculum and preparing your students for an immersive world.
    News
    June 9

    New release brings Mesh to MetaHuman to Unreal Engine, and much more!

    This release of the MetaHuman framework brings not only new features for MetaHuman Creator, but also an exciting new MetaHuman Plugin for Unreal Engine, as well as support for the new character rigging, animation, and physics features in Unreal Engine 5.
    News

    New release brings Mesh to MetaHuman to Unreal Engine, and much more!

    This release of the MetaHuman framework brings not only new features for MetaHuman Creator, but also an exciting new MetaHuman Plugin for Unreal Engine, as well as support for the new character rigging, animation, and physics features in Unreal Engine 5.
    News
    April 5

    Unreal Engine 5 is now available!

    With this release, we aim to empower both large and small teams to really push the boundaries of what’s possible, visually and interactively. UE5 will enable game developers and creators across industries to realize next-generation real-time 3D content and experiences with greater freedom, fidelity, and flexibility than ever before.
    News

    Unreal Engine 5 is now available!

    With this release, we aim to empower both large and small teams to really push the boundaries of what’s possible, visually and interactively. UE5 will enable game developers and creators across industries to realize next-generation real-time 3D content and experiences with greater freedom, fidelity, and flexibility than ever before.
    News
    December 9

    Introducing The Matrix Awakens: An Unreal Engine 5 Experience

    Get a glimpse into the future of interactive storytelling and entertainment with Unreal Engine 5 in this cutting-edge experience that showcases the start of an exciting new era for creators and players alike. 
    News

    Introducing The Matrix Awakens: An Unreal Engine 5 Experience

    Get a glimpse into the future of interactive storytelling and entertainment with Unreal Engine 5 in this cutting-edge experience that showcases the start of an exciting new era for creators and players alike.