Going Big: How Giant VR is Taking Cinematic Storytelling to All-New Levels in VR

Ask today's top content creators what excites them the most about the future of virtual reality and they might not be able to tell you. It's not that they don't have things about VR that excite them since the sheer level of immersion afforded by VR offers unparalleded possibilities for gameplay, storytelling, etc., but it's the fact that, as designers and artists, the unknown aspects of virtual reality are generally considered the most compelling. After all, we're are just now seeing technology begin to catch up with our wildest VR dreams.

For may of these content creators, it's not just about how we will transfer what we currently do to the medium of VR, but rather, how VR will transform the way we do what we do. How will virtual reality change gameplay? How will virtual reality impact multiplayer? How will VR help us tell stories and enhance immersion like never before? 

It's this last question that has driven the team behind Giant to pursue a project that is looking to take storytelling to all-new heights. Set within the basement of a home within an unnamed war zone, Giant gives viewers a glimpse into a world that is a harsh reality for some and a virtual awakening for others. In it, you witness a couple consoling their daughter while bombs blast nearby buildings and threaten their existence. As a film it would convey a sense of sympathy for the family that serves as its subjects, but as a UE4-powered VR experience, it delivers a connection that makes the experience deeply personal and emotionally exhausting.

After experiencing Giant during the 2016 Game Developer's Conference myself, I decided that I wanted to learn more about the project, the people behind it and their purpose moving forward. That's why I caught up with Director, Co-Creator Milica Zec, Producer, Co-Creator Winslow Turner Porter III, Lead Technical Artist Jack Caron, Director of Creative Technology Todd Bryant and Technical Producer, Depth Pipeline Designer Juan Salvo to discover how their influenctial project came to be and how it is now helping to shape the future of storytelling in VR.

For those who haven’t experienced the project yet, can you please explain what Giant is all about?

Milica Zec: Giant is a cinematic virtual-reality experience that premiered at Sundance Film Festival New Frontier in 2016. It’s a story about a family in a makeshift shelter hiding from falling bombs, in an unnamed war zone. In order to protect their young daughter from the very real terror that surrounds them, the parents invent a fantastical tale wherein the explosions they hear are actually the footsteps of a friendly giant who is approaching their building simply to play with little girl. The parents' fairytale intensifies as bomb-blasts draw closer and closer.

What were the various creative and technical elements that needed to come together in order to bring your vision for Giant to life?

Winslow Turner Porter III: In order to tell the story of Giant as immersive and compelling as possible, it was paramount for us to have real human actors with real human emotions inside the game engine. We wanted to avoid the uncanny valley at all costs. The live-action video was shot by DP Alex Corn on a green screen stage where the actors had about 8x3 ft to move. Making the video seamlessly mix with a 3D environment in real-time was a big challenge because no ready-made solutions were available when we first started production in September of 2015. We explored many different volumetric video capture systems, but there were still issues with jagged edges and occlusion. Also, our story revolves around a family who is struggling to survive in a conflict zone, so we needed a solution that could deal with multiple actors at the same time, often closely interacting with each other. Depthkit by Simile, which implements the Kinect 2, was the platform we ended up choosing because it was a highly effective and affordable capture option. Depthkit merged with Red Dragon 5k footage shot by Alex Corn is particularly powerful from a fixed perspective, because it only gives volume data to what is directly visible to the camera. Since we wanted our viewers to be seated on a chair which emulates the approaching bomb blasts it ended up working out quite well for Giant. 

When writing the script it was also very important to think of everything more like a one act play than a traditional cinematic short. There are no cuts like the audience is used to with traditional 16x9 media. The camera is fixed, pointing in the direction of the family interacting at the other side of the basement. The more we experimented with the actors and with Unreal Engine, the more it allowed us to really work out all of the elements that we could add to the story to help bring the environment to life while also maintaining a feeling of realism. One trick we utilized in particular was being able to swap out the video assets of the actors every time a bomb blast caused the lights to go out.  The best advice we can give to other filmmakers in VR is to experiment and take risks as much as possible, because it is such a new medium where the rules are created one day and broken the next. 

Did you always envision Giant with video featuring live actors or was the project considering alternate means of telling its story at some point?

Milica Zec: At first, Giant was supposed to be a conventional short film. After a lot of consideration about the purpose of the film, however, I realized that it would be best to tell this story through the VR medium. By inserting the viewer in the shoes of the family at risk, they can more directly witness and experience the fear and suffering that millions of innocent families around the globe are going through at this very moment. I wanted to transfer the emotion of being in a conflict zone to people who have never experienced such a trauma, so that they can relate to those to whom it is happening, and thus increase the chances that some kind of action might be taken. As well, we deliberately chose English-speaking, Western-looking actors - Jordana Rose, Zoë Winters and Clem McIntosh - to portray the victims, so that a Western audience could better connect. 

How big of a team did you have working on this project?

Winslow Turner Porter III: Before we started production on the project it was just Milica, Lizzie Donahue (the scriptwriter), and myself. As soon as we started at NEW INC, the New Museum incubator in New York City, we needed to quickly find talented artists who also were very tech savvy. People that were interested in pushing the boundaries of not only storytelling in VR but also the pipeline. Our timeline was based around checkpoints with Sundance New Frontier, who made their final decision at the end of November. In that period we partnered with our co-producers, Virtualize, and the team rapidly grew to over forty people. Because we are mixing live-action and Unreal Engine we essentially needed twice the amount of people. To us, sound design was just as important as the visual elements of Giant. We were lucky enough to work with the talented Serbian sound designer, Aleksandar Proti?, who, like Milica, lived in Serbia during the bombings, adding an extraordinary level of realism to the soundscape. 

We were also very fortunate that we had amazing production partners like Framestore and Hello World Communications and sponsors like HP, NVidia, Comcast Ventures and Microsoft. Since our premiere at Sundance we have grown even more, partnering with Technicolor and Scott Gershin to complete our sound mix, which added a final polish before we start to distribute on a variety of platforms.

Are there any films or gaming experiences in particular that inspired you to create Giant?

Milica Zec: Prior to making Giant, I had been following the VR scene for quite some time. One experience that made a particular impression on me was "Evolution of Verse" by Chris Milk. It not only moved me, but also gave me an impression of the possibilities and potential effects of VR media. 

Winslow Turner Porter III: I tried VR experiences in my adolescence at Epcot Center which were clunky and nausea inducing, but I still was totally enamored with its amazing potential. Fast forward to last year, 360 degree video content from Vrse was definitely influential in understanding how to tell stories in immersive environments. The Last of Us really showed me how emotionally engaging narratives in video games can be. We wanted something that could pull from the strengths of both. 

Jack Caron: For the environment work we were trying to get in a lot of realism to match the live footage acting, so we looked at the photogrammetry techniques used in games like The Vanishing of Ethan Carter. In terms of lighting and feel, Take Shelter had some great reference for our basement set. There was plenty of dramatic horror movie basement scenes to draw inspiration from, but we tried to make it seem as lived in as possible.

Todd Bryant: I come to game engines from a filmmaking background. In the five years I’ve been using them for my work I’ve actually never made a traditional game.  So, for me, I am very influenced by the interactive storytelling pieces that emphasize interpersonal connection and examine the human condition like Passage, The Graveyard and Journey and their carefully crafted aesthetic intended to elicit an emotional response from the player.

The creative in Giant draws heavily from your personal experiences as a child, Milica. Beyond the obvious sense of visual immersion, how do you feel that VR presents a new, richer capability to connect viewers with situations and circumstances that they may or may not be familiar with?

Milica Zec: Giant is a fictional story loosely based on my experience growing up in Serbia during the bombing of Belgrade. I shared my family's experience with Lizzie Donahue, the scriptwriter of Giant, who created a story with American family in conflict zone, to which a Western audience could relate. Our reasoning was that if the family looks, speaks, and dresses like them, viewers can connect more with families at risk, and this would hopefully bring more compassion and understanding towards people in these kinds of situations. 

As a cinematic experience, Giant breaks new ground by leveraging VR to enhance immersion. With this in mind, how do you feel VR will impact storytelling moving forward?

Jack Caron: We like to describe Giant as VR theater. The actors play out the drama in front of us, and there is still a fourth wall there, but we occupy the same space. It creates a direct, undivided emotional link between the viewer and the cinema that is difficult to replicate elsewhere (possible exceptions being semi-interactive theater). That is the power of VR. As we craft these experiences it is important to understand the idea that timing and cues are more akin to a choose your own adventure book than they are the linear flow of traditional cinema, so the storytelling will need to embrace that.

Do you feel that long-format versions of VR experiences like Giant are the next step in storytelling? What limitations of the medium need to be addressed before we can move forward with long-format, cinematic VR?

Jack Caron: A huge part of storytelling is about drawing the viewer (or reader) into a world, and VR lends itself well to that. Will it be a replacement to traditional storytelling? Probably not. But yes, as the technology and techniques get defined and refined to where more people can create in VR it will be the next step in immersive storytelling. In terms of the physical aspect and limitations, this question comes up quite a bit now that there is a lot of interest in episodic and feature length VR. Headset fatigue is a big (and somewhat person specific) concern right now. As the hardware becomes lighter and ventilation becomes a focused improvement, I think the experiences will start to expand in length. 

One particularly unique aspect of Giant is the seated, multi-person experience that it offers. At GDC 2016, I witnessed Giant alongside two other people, which brought a shared experience to life for us all. Were there any particular technical challenges brought on by this design choice and, creatively speaking, why was this set up an important aspect of experiencing Giant? 

Todd Bryant: Giant is an emotional VR experience and we felt that it was better to watch with other people in the room.  While the viewers obviously couldn’t see each other we used open-ear headphones so that they could communicate with each other verbally allowing groups of friends and families to share their growing concern for the family depicted in the experience and reassure each other. This added the technical challenge of keeping the experiences perfectly in sync with each other otherwise the audio from the other headphones would interfere with each other and the haptic feedback provided by our rumbling seats would ruin dramatic moments.  We were able to overcome this hurdle by running everything using Open Sound Control over a private network.  We triggered and reset the experience via a custom iOS app that controlled the show controller computer that relayed commands to the three production computers running the headsets and triggered different images on our projector.

Aside from having a rich impact on the viewer, how do you hope that VR experiences like Giant can generate awareness about serious issues throughout the world in pursuit of positive change?

Milica Zec: VR is a powerful tool that can transport you to new places and situations and let you experience things you’d never imagined would be possible to experience first-hand. It can broaden our horizons and bring a new understanding of the world around us. And, if used wisely, can be an important and effective vehicle for social change. 

You’ve exposed a variety of audiences to Giant through different events including Sundance and GDC. What has the feedback been like from people who are experiencing it from the various creative perspectives?

Juan Salvo: The feedback from people experiencing Giant has been tremendous, and tremendously rewarding. Often when working on innovative technology like this, you don’t know how audiences will experience it. At Sundance we saw people moved to tears by the story. VR as a platform has the potential to be much more immersive and emotionally engaging than traditional media…here we were able to harness that to great effect. And watching viewers react in real-time was immensely rewarding.

Todd Bryant: One of my favorite experiences and opportunities to get incredibly insightful feedback was at Sundance during the Filmmaker’s Evening. Giant was installed at New Frontier which is the funhouse of Sundance where they often throw parties in the evenings. One evening they had a party just for directors with films at Sundance. Many of the directors had never tried on a head mounted display before and Giant was their first VR experience. Many were exited about our experience, noting that due to the immersion provided by the headset we had done in 6 minutes what would take them 45 minutes to an hour to do in their own medium. Most were giddy about the possibilities.

Jack Caron: At Sundance the idea of cinematic VR really was very exciting to a lot of people that approached the booth. They ranged from actors to directors to people just walking in looking to see something new. The feedback was filled with genuine intrigue after seeing live actors filmed and integrated within a game engine. GDC brought a whole different technical aspect to the feedback. There were far more conversations regarding our methods, including both the software and hardware, as well as questions from a lot of like-minded developers looking to do cinematic VR.

What technical capabilities of Unreal Engine 4 helped bring your vision for Giant to life?

Jack Caron: With very limited access to hard coders on the project we leaned heavily on Blueprints to get the project running and adaptable. Sound, dynamics, level loading and level streaming were all handled via the node editor, as well as some scalability and Matinee control.

Todd Bryant: Giant was built in 4.9 and is run by a series of Matinee nodes. I am excited about the possibilities provided by Sequencer in the newer engine releases and how that will change the development process for the next wave of cinematic releases.

Did you discover any technical tricks or tidbits throughout the development of Giant that might be beneficial for other Unreal Engine developers to know?

Jack Caron: Using custom event dispatchers and public game state variables to communicate between levels was a technique we used after separating different aspects of the experience to streaming levels. To use the depth data we created a disparity shader that allowed us to separate the live footage into left and right eye playback, using one main video file with side by side composited data.

Giant was an ambitious project for your team. Where did you encounter the most challenges and what have you learned throughout the process?

Jack Caron: There are several things, as with anything that really has not been done extensibly before, that we have learned from when trying to get live footage into the engine set. Some include our filming process; lens focal distance and actor/environment interaction (while filming on green screen). Others focused on engine performance and keeping our frame rates up while not cutting quality and lighting. One main challenge was developing the footage depth and tuning that in. With the Kinect 2 and Depthkit data we had a few avenues in which to try and create a 3D effect. It took a bit of trial and error to reach the final disparity shader, but we had tried everything from parallax offsets to displaced geometry.

What’s next for Giant and for your team in general?

Milica Zec: We are planning to create a trilogy consisting of three short virtual reality experiences that examine the current state of our collective life on this planet. Through three discrete yet connected narratives, “Trilogy” explores some of the world’s most pressing current problems: "Giant", the first installment, speaks to atrocities that humans commit to one another. Part 2 (unreleased) will cover atrocities that mankind brings upon Mother Nature and her non-human inhabitants. Part 3 (unreleased) offers a hope to our dilemma, presenting a solution of a sort to the devastating predicament in which we’ve left our planet and ourselves, as it speaks of things larger than our temporal existence.

Winslow Turner Porter III: Giant will be released on most of the major video game platforms that support head-tracking HMDs. We will also be distributing a 360-degree stereoscopic video of the experience for smartphone based viewers, so that we can reach a larger audience.

In addition to the Trilogy, we are very interested in continuing the exploration and experimentation of how to make storytelling as immersive as possible. We have been talking to a variety of different clients and collaborators about projects ranging from music videos to episodic dramas. Milica and I first started collaborating together 9 years ago, making music videos and short films, so it feels very natural to extend it into this ever evolving medium. 

--

For more information on Giant VR, please visit the official website.