May 8, 2019
Virtual production: motion control and real-time preview at Stiller Studios
In this podcast hosted by fxguide’s Co-Founder Mike Seymour, Forsberg describes how Stiller Studios helps old-school directors work in a “new-school” environment, giving them the ability to see their live-action footage perfectly in sync with their CG imagery, in real time, on set. You can listen to the full podcast below, or read on for an overview.
Stiller Studios focuses on intricate motion control work, where virtual and real camera positions and paths need to be perfectly matched and output in real time as usable data. But how did this all start?
In the early days, Forsberg was grappling with getting the perfect green-screen shot. He went to Mark Roberts Motion Control in London to purchase a small robot for moving his camera a meter or two, and came back with a Cyclops motion-control rig weighing in at 4.6 tons and with the ability to reach up to six meters in the air.
“They had this giant beast standing right behind me, and I guess it was something that they thought of, because your age doesn't really matter—you're always gonna be a boy,” laughs Forsberg. “And I got home with that 4.6-ton unit.”
Once the Cyclops was installed in the studio, it was initially hooked up to some Autodesk products. However, in the last five or six years, Stiller Studios started working with real-time engines, and today the motion control data goes directly into Unreal Engine. “It's like having a 16-meter track with a six-meter crane on it, but it sends out FBXs and camera positions, lens data, everything, so you can live-mix it in Unreal,” says Forsberg. “You can see live action on top of computer-generated imagery in perfect sync. What we want to provide is a way for the director to see the full picture [...] on the screen, and that includes light directions, shadows, interactions with everything.”
Real-time on-set review
In order to see that full picture, directors are given an iPad so they can walk around the virtual set and see it fully rendered in UE4. They can even use the iPad as a virtual camera, recording a camera path that can be precisely repeated by the real camera mounted on the Cyclops. “We let the director or the DOP move [the iPad] on the set; he will see exactly how the camera is going to move, and then—only seconds later—the 4.6 ton beast does that exact move again,” explains Forsberg. “After 11 years of doing this, it's still cool to see.”This repeatability is a key factor in pulling off some of the more complex shots the studio undertakes, and was used to great effect in the award-winning short film Echo by Victor Perez. The time-bending, bullet-time-like mirror sequences that give the film its unique character required shooting the same subject from different points of view on the same set and then combining them. For this, Stiller Studios employed a second high-speed motion-control rig, the Bolt, alongside the Cyclops.
“You split the bullet time in some kind of 3D depth, so you would have one camera that gives you live action, the other one would give you bullet time, and they would swap back and forth,” explains Forsberg. “So one of the time layers would bend, but within one frame, and that's only doable because of the motion controls, because of us being able to program them so they go back and forth in time and in camera speed, and then ultimately sync up again.”
Digital doubles: recreating technical setups in UE4
In addition to the two camera rigs, Stiller Studios also has a computer-controlled motion base for actors, capable of taking up to one ton. This is useful for simulating movement of live actors in a helicopter or plane, or a real person riding on the back of a CG creature. For this kind of shot, UE4 is also used to visualize the technical aspects of the setup, with digital doubles of the rig and camera in the engine.“To us, Unreal is very much a techvis, an on-set-vis, and a previs tool,” says Forsberg. “If we put a flying car in there, or a flying carpet, then we can play around with it in real time. We have a miniature copy of the model mover, so we can play around with it like a six-axis joystick and move it around, and we can see that live in Unreal as well.”
While this highly advanced setup is intended to be used for serious work, the team at Stiller Studios can’t resist having a little fun.
“It's one of the best toys in the world,” says Forsberg. “Late Fridays, you can sit on top of that motion base and be flying an F16, or whatever. When you hook it up to the Unreal Engine, it's like you got the biggest flight simulator in the world. It's the same technology that you would use for big airlines and everything, but we've got it hooked up to the film system or to the camera system.”
It’s not just the rig and the camera that have digital doubles. Stiller Studios has also made virtual copies of their approximately 30 RGB LCD lights. “We can match the 3D and the live action, and that's really helpful for DOPs and for directors to get a sense of what's going to come out,” says Forsberg.
Perfectly synced output to post
While the real-time output from some projects is used as final pixels, for many others the on-set experience is more about getting the shots nailed, and there’s still more work to be done in a traditional VFX post-production pipeline. Making sure the assets transition usefully to that final stage is critically important, and perfect synchronization is key.
“Let's say you have the camera on the motion control and then you get your FBXs out of your Unreal,” says Forsberg. “You might be working with virtual actors as well, using the motion capture system. What we take pride in is delivering files that sync perfectly together so the artists can be artists [...] and their job will be to make it more beautiful.”
Greater creative freedom
Among the advantages of the approach that Stiller Studios is taking, because the teams are able to actually visualize the final shot on set, it frees creatives up to experiment, and enables actors to be more spontaneous.
“It makes the creatives braver, which is good for the film because, back in the days, you would take something that you would be sure would work later,” says Forsberg. “Now that you get the real-time feedback, you can actually test things and push limits.
“To me, a shot is so much more than the actor in it. It's the way the actor interacts with the composition and everything. And that too can be decided upon on set. One of the most important places to have the VFX person is on set, because that's where we make decisions and that's where we try to do the right thing.”
Loving the challenges
While Stiller Studios makes combining all of this complicated engineering look easy and fluid, it’s no mean feat. For example, as Forsberg says of calibrating the equipment, “I had one of my technicians doing some calculations, and he told me there were fifteen point something million ways of doing this wrong, and only one way to do it right!”But Forsberg shows no sign of losing his enthusiasm for the challenge. “After 11 years of doing it, it's still like going to the playground every single day,” he says.
This podcast interview with Patrik Forsberg is part of our Visual Disruptors series. Visit our Virtual Production hub for more podcasts, videos, articles, and insights.