Image courtesy of Paramount+

Why PXO created a real-life ‘Holodeck’ for ‘Star Trek: Discovery’

In the fourth season of Paramount Plus’ Star Trek: Discovery, the show boldly goes where no previous Star Trek series has gone before—both on screen and behind the lens. 

Since the series debuted in 2017, award-winning visual effects and virtual production house Pixomondo (PXO) has elevated the visuals in the latest iteration of Star Trek to feature-quality work. Audiences and critics alike have lauded the designs, garnering the studio a 2021 Emmy nomination. PXO also helped JJ Abrams and crew earn a Best Visual Effects Oscar nod for its work on the 2014 film Star Trek: Into Darkness, so it’s fair to say the studio knows how to make the world of Star Trek look authentic.  

But despite the established success, for the most recent season of Star Trek: Discovery, PXO went in an entirely new direction. Rather than sticking with a proven formula built around blue and green screens, the VFX house instead embraced virtual production, starting with a massive new custom-built set featuring dozens of LED panels, motion tracking cameras, and Unreal Engine. 

Or as some would say, a real-life Holodeck.
 


Holodeck engaged

When Star Trek: The Next Generation debuted in 1987, it introduced an iconic (and sometimes almost magical) location known as the Holodeck. Created as both an entertainment and research tool, the Holodeck could create entire worlds in a single room. At the time it was considered to fall more on the “fiction” side of the science fiction series, but that was before virtual production. 

That’s where PXO’s new virtual production stage—nicknamed the “Holodeck”—comes in. Located in Toronto, the stage —created in conjunction with William F. White Int'l— consists of roughly 2,000 LED panels on the wall, and another 750 panels on the ceiling, and measures 72 x 85 x 24 feet. More than 60 OptiTrack cameras surround the stage to provide tracking, and it is all run by more than 40 high-end GPUs running Unreal Engine’s nDisplay to synchronize the 2,750+ panels. The stage can’t conjure a Klingon bat'leth out of thin air just yet, but give it time.
Image courtesy of Paramount+
The set itself is also adaptable, allowing for the inclusion of real-world props and set trimmings. So when the performers arrive on the stage, they are seeing highly polished visions of alien worlds and futuristic locales to immerse themselves in. While operating it isn’t quite as simple as hitting a button and ordering up an interactive mystery in Victorian England, or creating virtual water so you can kayak down an alien river, PXO’s virtual production stage and “AR Wall” may be the closest thing around to a real-life version of the original Holodeck. 
“There are many components to the AR system,” said Frank Siracusa, Executive Producer of Star Trek: Discovery and Star Trek: Strange New Worlds, “but the real magic comes from the Unreal Engine software used to project 3D images onto a wide surface, providing scope both on the screens and in the volume or ‘footprint’ provided to stage the scenes and populate with set pieces and performers. 
Images courtesy of Paramount+
“Combined together, these components offer the filmmaker the ability to execute scenes that would otherwise require a film production crew to travel to distant locations, and some cases, into outer space.”
 


Exploring the 32nd century

To help inaugurate its Holodeck in style, nearly every episode of Discovery’s 13-part fourth season has at least one scene shot against the AR wall. Environments ranged from familiar locations like the Discovery shuttle bay, to fantastic locations like the Kaminar Council Chamber, located deep underwater.
 
Video courtesy of Paramount+

To create each new environment for the AR wall, the process begins in pre-production, with PXO’s Virtual Art Department (VAD) creating prototypes for each environment. To quickly create a rough representation for blocking, the team sources elements from the Epic Games Marketplace, adapts them as needed, and adds them to the scene. 

From there, PXO then breaks the design down to distribute it to a larger team of artists, all collaborating to create the build for the final environment within Unreal Engine. After that, the production team—including the director of photography—can make any changes they want. Then roughly a week before the shoot, artists apply the final touches. There may be a little post-shoot touchup work (most often around the seam between the ceiling and wall), but otherwise the work is already done.
Image courtesy of Paramount+
“Unreal Engine has played a pivotal role throughout our entire virtual production and VFX pipeline at PXO,” said Mahmoud Rahnama, Head of Studio and VFX Supervisor at PXO. “There really is no competition in the real-time space that integrates so well into existing industry standard packages. Unreal Engine seemed like an obvious choice.” 
 

The hidden benefits of virtual production

To fully integrate Unreal Engine alongside industry-standard tools like Maya, Houdini, Photoshop, Substance Painter, ZBrush and Nuke, PXO decided to rethink its approach to asset creation. Initially, artists would create two sets of assets: the first, a high-resolution version made with legacy VFX tools; the second, a lower-res version created in Unreal Engine. After a bit of brainstorming, PXO found that there was a better solution: go all-in on Unreal Engine. 

“Using a procedural approach directly in Unreal Engine ended up being the better approach,” said Rahnama. “Not only did this allow for much higher-resolution assets in UE, it yielded similar results to our offline render, but also dramatically reduced our average offline render time.”
 
Video courtesy of Paramount+

Along with the obvious advantages, PXO discovered several hidden benefits of using Unreal Engine for virtual production. PXO was able to totally abandon its offline rendering approach to cleanup work, one of the most thankless—but necessary—parts of any VFX artist’s job. That also created a domino effect, freeing up other groups across the board.

“It's great being able to focus the team's attention on the shots they feel excited to work on rather than some of the more mundane work,” said Rahnama. "That is one of the many creative benefits for artists working in virtual production."

The lighting team especially saw a major shift. During the first season of Discovery it could take up to 14 hours for a lighting bake to finish. Now, using Unreal Engine’s GPU Lightmass Baker, high-quality bakes can be done in as little as 30 minutes.

One of the biggest, most important changes, however, is somewhat intangible. In previous seasons, each location would be divided into multiple tiles, and individual tiles would then be assigned to an artist. Once the tiles were complete, they were reviewed independently and the complete environment was only assembled for the final batch of reviews. During Season 4, PXO’s artists could all work in the same scene simultaneously. That not only led to a more cohesive and consistent environment, it also enabled artists to collaborate and inspire one another. Plus, as an added bonus, when artists know people will see their work on a daily basis, it tends to significantly improve organization. 
Image courtesy of Paramount+

The next frontier

For PXO and its Holodeck, the docket is full. With work complete on the fourth season of Star Trek: Discovery, PXO began working on the new Star Trek: Strange New Worlds, which debuts May 5, 2022, and also employs virtual production. It is also working on the latest season of the hit superhero show The Boys, the video-game-turned-TV-series Halo, the Game of Thrones spinoff House of the Dragon, and a handful of feature films. But PXO also has its sights set on a future where entertainment is just one part of a much bigger, connected whole.

“We are in an interesting time when it comes to 3D content creation, and all industries are realizing the benefits around it and are quickly adopting it into their products,” said Rahnama. “We are currently refining our workflow for television and film, but can easily see our assets and tech being used in many different industries. 

“Right now, the speed at which you can shoot a sequence and cut it together with 80% of the shots completed and not have to wait months to see final VFX is a game changer. Soon, it’s going to be a standard way to get everything out the door, and onto people’s screens, even faster.”
 

    Get Unreal Engine today!

    Get the world’s most open and advanced creation tool.
    With every feature and full source code access included, Unreal Engine comes fully loaded out of the box.