HOW THE PROJECT STARTED
Some months before officially beginning the project, the idea of making a short movie as our final project at NAD School was already in our minds. It was during the holiday vacation that we really decided to go forward with it and we worked hard to make a solid storyline before starting school in January. Being in a Video Games school program, we knew that working with tools like storyboard and animatic would be a challenge for us because we hadn't used them as much as our FX/Cinema colleagues.
It was necessary to plan all the shot angles and the cuts before starting the character animations. It is not possible to bypass these steps because when you are making a cinematic, each further modification affects a lot of things in multiple ways. Adding just one second to the shot requires a lot of modifications, especially when you have a lot of track in the Matinee.
So in the first two weeks, we made an animatic in Maya with an approximated camera to validate the timing and the compositions. Then, by the end of January we successfully had an in-game animatic in Unreal. At this point a lot of assets were only placeholders.
Inevitably, a lot of shots changed when the final animations were starting to be implemented and the facial expressions integrated.
In the last month of the production, we started to polish everything and we did a first pass on the sound design / music. On the art side, there were only a few tweaks needed.
Our animation workflow was pretty straightforward. We animated the characters based on the cameras of the animatic, and then exported them relative to the center of the world from Maya to UE4. We did this to ensure that all of the animation was in the good place. We used a lot the UE4’s export features to get the correct scale and work our animation based on it.
Almost every shot had its own animation sequence for the characters, with no physical continuation between them. This enabled us more freedom of shot composition. We had only a single Matinee where we stacked all of the objects and the character animations, along with events controlling the loading of the levels being streamed. At the end we had almost 200 tracks, and the engine was still running smoothly at runtime.
The lighting setup changed many times during the production. At the beginning, we were looking for a darker and grey feel, just like the feeling before a thunderstorm. This was to give the Miss Daisy diner a more mysterious and threatening look. As the character’s development became cartoonier, it became clear that we wanted a more saturated color palette.
For the lighting colors, we looked for some variations of cool and warm to emphasize the mesh volumes and give a more stylized look, but we didn’t want to have a lot of contrast in order to avoid getting too dramatic. Therefore we chose a more diffuse and soft look for our environment lighting. This was achieved by tweaking a lot of parameters of the Lightmass tool, like the Static Lighting Level Scale.
The main purpose of the lighting in our project was to direct the viewer’s attention to the key elements of the story, mainly the characters. Since these assets are Skeletal Mesh, special attention had been given to the volume light sample that Unreal generates. Since this system does not create precise lighting for the purpose of optimization, we used numerous dynamic lights to hide the artifacts. The main challenge was the clear difference between static and dynamic mesh when they were side by side.
We ended up using dynamic lights to add more crisp and dramatic shadows on the characters.
Another challenge for our project was to handle exterior and interior environment in the same level, while maintaining the cinematic quality. It was hard because shots were cut directly from the exterior to the interior (2 different levels), and this caused texture streaming issues and mesh popping.
We had to manage the lightmap size to accelerate the loading time and limit the resolution. This would not have been a problem if the video was captured frame by frame, but we really wanted to capture the entire film in real time to better highlight the power of UE4.
To give a more interesting look to some shots, we faked casted shadows from hidden objects.
Inside the diner, the lights coming from the exterior are not generated from the directional light. Instead we used a combination of static lights to fake the bouncing of the exterior light inside the diner. Then, we used dynamic lights to create a sharp shadow of the sun blinds.
The light coming from the rooftop shares the same blueprint so the modification of multiple light is very fast!
For visual effects, we first looked at what was done in animated movies and video games. We noticed that in the current generation pre-rendered visual effects for smoke and fire, fluid simulation are often used to achieve quality and depth. We wanted the same level of quality for this project. We also wanted to keep a certain degree of softness in the VFX details in order to avoid a big contrast with the characters. To pull it off, we simply built a light rig consisting of RBG colors to render out a normal map.
With normal map.
Without normal map.
After that, we did a lot of research for references. It was very important to have a reference bank as well as an influence bank because we were able to learn a lot from what has been done. Having references was very helpful for production speed, while the influences helped us create a specific style and learn how to achieve certain looks.
So after our artistic vision had been set, and after doing all the research, we started the simulation for the effects that were the most important and risky. We did our simulation with Fume FX and Realflow. Making simulations can consume a lot of time and reduce precious production time, especially for an academic production. It was also important to have a good knowledge of the simulation system that we were going to use, because we could have lost a lot of time with an effect that would not make it into the scene. We learned that the reutilization of assets is a time saver!
Once the simulations were done, it was time to integrate them into Unreal Engine 4. UE4 has an excellent material editor as well as an excellent particles editor. There are a lot of documentation and tutorials online that helped us implement the result we wanted to achieve. To get the simulation to work in a game engine, we needed to create a sprite sheet. Thus, we could animate the simulation frame by frame and it was a super easy to set up with UE4. An easy way to create sprite sheet is to use the free Photoshop plugin, Contact Sheet II.
Sprite sheets look good, but they generate too many draw calls and overdraws that come with visual effects. So it was important for us to have a plan B and we had created a simple smoke material that had a single texture call with 3 different smoke patterns in each color channel. Then we used two of the color channels to create a UV Distortion effect on the third channel. We added some control over the tilling and the distortion strength so we could create more nuances in different smoke situations and have a very low draw call that drives a lot of effects. There is a lot of information on the web for those kinds of effects in materials.
Another trick we learned was to use meshes! They can become your best friend in certain situations where you need to create a fake volume, to cover a large surface or to create a fake volume light. We used mesh for a fake volume light and for some smoke that covered a large area.
We also wanted to use as many GPU particles as possible as they were better for performance. They also allowed us to create special effects with as many as 50,000 sprites utilizing a vector field. Here’s an example:
In the end, the inevitable returns: optimization. We optimized many visual effects by removing certain elements as well as significantly reducing the sprite sheet size to improve the texture draw call and reach the desired performance.
For this project, we used a particular technique to do all of the destruction. In order to have a persistent destruction animation across different shots, we didn’t use the in-editor physics engine because of its random behavior and performance cost. The solution we came up with was to bake the destruction directly in 3DS from a MassFX simulation. We then transferred the animation onto a skinned mesh with an animation baking script made for the project (available here). MassFX enabled us to have a lot of control of the physics animation of our scenes, because of all the parameters available in the plugin.
When the animation had been baked, we imported it in the same way as a skeletal character and created a track in Matinee to play the animation.
When we first started working on Miss Daisy, the environment team established two distinct objectives. First, we wanted to visually produce an impressive short while understanding the limits of real time and its optimization. Second, we also wanted to emphasize the storytelling through the environment.
To clarify the story, and also push the character development, it was important for the team to emphasize the narrative side of the environment as much as possible. With only a few minutes to tell a story, it was helpful for the artists to visually show as much backstory about the past and the character of the old lady as possible. The diner was the perfect place to showcase her twisted and dangerous sides by displaying various weapons as wall mounts, or random decorations. By creating so many different weapons, we were able to craft a certain mystery surrounding the old lady, while emphasizing a light and cartoon feeling to the comic short.
During our first month, which was dedicated to the pre-production, we established what artistic direction we wanted to take with the help of multiple moodboards, reference documents and concepts. For our first environment, we decided to recreate a stylized version of the well-known giant sequoia forest. We were attracted to both the impressive scale of the trees, and the unique color palette of the vegetation. The second section of the environment was Miss Daisy’s diner. We decided to do an old restaurant inspired by typical 1950’s diners. Considering the diner’s remote location, we gave the building a few cottage elements by adding more wood and staying away from chrome and urban materials.
The environment was created based on every distinct camera shot in the comic short. We had to analyze each camera angle in order to create interesting and diverse compositions. The level art was an iterative process, where we would work on each composition multiple times with the arrival of new assets and lighting modifications. Each camera shot had its own lighting setup to help focus the center of attention on the characters or the action, and also to embellish the different shots.
Unreal has a great workflow for landscape creation because of its compatibility with World Machine (a terrain creation tool). We first created a base landscape in Unreal with the built-in tool. Then we exported it in RAW16 format which we then imported in World Machine. We created a macro to simulate the erosion process in order to have a similar style for all our landscapes. The macro exported back a heightmap file along with an erosion map and a flow map that we imported back into the landscape shader which gave us more control over the color in these areas.
The landscape shader adapts itself relative to the slope and elevation of the mountains to add snow, rock or grass to the terrain.
The optimization was a big part of the project because it was very important to maintain framerate without dropping below 24fps. Below 24fps, the spectator will notice artifacts and lose focus from the animation. In a video game, it’s less annoying because the player will focus on action flow, but in a short movie, the spectator is passive and he is more likely to notice it. Also, in order to prove that real time is an efficient way to make this kind of project, the fluidity must be the same as a pre-rendered movie.
Having to deal with an interior and an exterior in the same scene was a big challenge. To avoid impacting the performance too much, we cut our scene into multiple stream levels and loaded them strategically when needed. We used the visibility track in Matinee a lot to hide cost-heavy objects like the skeletal meshes.
For heavy meshes we used up to three levels of details, especially for the trees and foliage. We managed to limit the popping effect of the LOD by slowly moving the exterior camera.
David Bastien Robert, Environment Artist
Francis Bouthillier, Technical Artist
Patrick Chamberland, Animator
Samuel Gauthier, Environment Artist and Lighting Artist
Alexandre Grenier-Marcil, FX Artist Christophe Lessard St-Gelais, Environment Artist
Medhat Hanbali. Music Composer
Valentino Lercher, Audio designer
Supervisors: Jocelyn Benoit, Marc-André Houde
Nadial El Mokhtari, Character Artist
Erwan Jonathan Mayel, Character Artist
Michel Quach, Character Artist
Edwin Jang, Concept Artist
Raphaël Nadeau, Concept Artist
Jahanzaib Chughtai, Rigging