21. November 2019
The Coalition dives deep into the tech of Gears 5
We caught up with the studio to discuss how they leveraged Unreal and source-code access to deliver best-in-class post-processing effects, particles, and volumetric lighting while achieving 60 fps on the Xbox One X. Speaking to how they achieved smooth and consistent performance across a wide variety of hardware platforms, they share how they elegantly leveraged tools like HLOD to allow more draw calls, which enabled them to develop the game’s highly detailed and diverse environments.
Considering Gears 5 features some of the highest fidelity character models seen in a game to date, we also discuss how they used dual lobe specularity, backscattering, and sophisticated eye rendering features coupled with leading animation techniques to deliver highly believable digital humans. The Coalition also discuss some of the challenges they faced developing the game’s open-world segments, which are new to the franchise, and explain how they overcame these hurdles.
Watch "The Visual Technology of Gears 5" presentation from Unreal Dev Days 2019.
With this being The Coalition’s second Gears game, how has development changed with Gears 5 compared to Gears of War 4?
Campaign Design Director Matt Searcy: Gears of War 4 was really focused on proving we could make a great Gears game. It was important to us that fans saw The Coalition carrying the torch for the world, characters, and gameplay. A lot of our focus on that game was around gameplay hitting parity with Gears [of War] 3 (with a few tweaks) and developing new Gears combat, enemies, and modes around that core experience.
In that sense, Gears of War 4 was really pre-production for Gears 5. We had confidence in our ability to make a great Gears game, and from the reviews and feedback we received, it was clear we hit that goal, and that the community wanted to see it pushed further.
On Gears 5, we started with a directive to challenge expectations. It still needed to feel like a great Gears game, but our pre-production was focused on exploring new experiences and how they could fit with Gears. Features like Jack, the Skiff, and Escape pushed the design of the game to new places, while other disciplines were able to invest in new techniques and technology from the start. Without the foundation and expertise we developed on Gears of War 4, we could never have explored these in the time we had.
Gears 5 features an impressive array of visual effects that include per-object motion blur, screen space reflections, sophisticated depth-of-field effects, tessellation, and more. How did the team employ the elegant use of these post-process effects?
Studio Technical Art Director Colin Penty: We employed these post-process effects very tactfully as we didn't want to compromise visual quality but still wanted to maintain 60 fps on Xbox One X. We integrated all the post-process effects we could from Epic's latest UE4 release into our own version of the UE4 engine, with the new Diaphragm DOF being the last post-process integration we did.
In terms of how we used these post-process effects, we generally used the non-glossy high quality SSR in campaign wherever we could afford it on Xbox One (PC does glossy SSR). For tessellation, we had our own custom async tessellation shader we created that we used almost exclusively on our snow and sand Materials. If you want to deform sand and snow dynamically around the player, tessellation is almost required to do that realistically unless you've pre-tessellated your mesh extremely high. Bloom was used liberally throughout the game, though we generally tried to avoid the tightest Bloom kernel as that one is quite expensive on GPU performance. We incorporated more camera exposure swing than Gears of War 4, allowing the camera to adjust to bright and dark areas of the map, though again being mindful of performance if we had areas of the game that didn't have a wide range of lighting values, we would lock the camera exposure to improve performance. Finally, we did a tuning pass across the game near the end for lens flares and lens dirt.
We also set up our post-process chain to run asynchronously over the next frame, which allowed us to claw some performance back. This made a lot of sense given our post-processing pass was quite expensive due to always outputting at native resolution. We leaned on UE4's temporal upscaling to scale the internal resolution of our base pass/lighting/translucency/etc. to maintain GPU performance.
Gears 5 uses impressive lighting with volumetric lights and fog coupled with dynamic shadows. How did the team incorporate this?
Penty: Our team was a big fan of the Volumetric Fog system added to UE4 and used it wherever we could. We created a Blueprints system that allowed us to place volume fog primitives throughout the level, which we will be discussing more in-depth at GDC 2020. I'm proud of the volume fog quality we were able to achieve in the campaign while maintaining 60 fps on Xbox One X.
For lighting, we made the decision to get away from using stationary lights, which we used a lot of on Gears of War 4. This allowed us to stop using shadow map textures for shadows saving us texture memory, speeding up our shader rendering, and allowed us to have full real-time shadows. We instead used Moveable Lights for all of our lights in the game, with a small tweak where we allowed Moveable Lights to bake out indirect data using Lightmass. This all gets stored in our lightmaps, which are different than shadow maps in that they store all the color GI data, not shadows.
Beyond our Directional Light shadow cascades, we used distance field ray traced shadows to shadow the environment. This is where the lighting system used a distance field representation of the mesh as the shadow caster. Since this was only seen in the distance for us, we could afford to have the distance field representation not be super accurate to save some memory.
Of course, it takes a talented lighting team to pull together all these technical elements and deliver some great looking visuals, which our team did an amazing job of.
The game features fantastic particle effects. How did The Coalition deliver on this front?
Penty: Our VFX team delivered some amazing effects on Gears 5. This was especially challenging as consoles don't have the strongest CPUs, and VFX can really be taxing on a CPU. To get around this limitation, we created a new VFX system using the Material Editor called Swift Particles that allowed us to create VFX using vertex offsets in a shader that didn't have any CPU overhead. We primarily used Swift Particles for environmental VFX [such] as snow, dust, and rain.
We made use of GPU particles wherever we could as well, though for certain more complicated effects, that was difficult to pull off. We added a "GPU Spawn" ability to UE4's particles that would allow the particles to spawn on the GPU as well.
To keep our CPU performance in-check, our tech art team spent a lot of time working with VFX to ensure the particle counts and emitter counts were reasonable while also making sure we used Particle LODs. We essentially had to go back through our Gears of War 4 character and weapon VFX and re-optimize everything for Gears 5 as Gears of War 4 was a 30 fps game.
The character faces in Gears 5 are fantastic with elegant uses of subsurface scattering and dual lobe specularity. Increasing fidelity even further, eyes also feature realistic light scattering. How did The Coalition manage to create the game’s beautiful digital human faces?
Penty: Our character rendering and performance is probably one of the larger visual jumps over Gears of War 4. Once development on Gears 5 began, we immediately went to work on our character skin, eyes, and hair.
For our skin, we implemented dual lobe specularity like you mentioned, which greatly improved the quality of our skin "sheen." We also refined our skin backscattering using photographic reference for ground truth. We added detail normals for pore detail as well.
For the hair, we created a better tool for generating hair AO using Houdini Engine in Maya to better ground the hair. We did some tuning of the hair Material but kept the shading model the same for the hair as what Gears of War 4 used, which was a Disney Marschner shading model.
We probably spent the most amount of time on the eyes. We added the Paragon bump offset to the Iris, we switched the eyes to using SSSSS (screen space subsurface scattering) so they looked much more natural in the socket (Gears of War 4 did not use SSSSS on the eyes). We added a lot of additional eye geometry such as eye water, tear duct, and eye AO geometry, which much improved the realism. As a finishing touch, we added in a dynamic Iris Caustic system inspired by Jorge Jimenez and Javir von der Pahlen's GDC 2013 talk. The faces were also re-worked to have much more accurate bone and face structures, which helped the believability a lot.
Art Director Aryan Hanbeck: We really wanted to make a significant improvement on character faces from Gears of War 4. The very first few seconds of the game feature a very closeup shot of Kait's face waking up from her nightmare. We had a small strike team working on Kait’s face for most of the project and they used that close up shot as the proving ground to show progress. Every once in a while, we would see a big jump and would be happy with the results until we would find something else to improve. The nice thing was that as improvements were made to Kait, we would then be able to carry those changes over to the other characters, so the entire cast made a quality jump.
Gears 5 is being praised as arguably the best game to take advantage of HDR. Can you elaborate on how the studio leveraged machine learning to pull this off?
Penty: We used a machine learning algorithm that was developed by Redmond's ATG group to train an inverse tone-mapper for color-space conversion. We then blended this 50 percent with a Reinhard buffer to allow maximum control.
The technology behind the HDR is only part of the equation though. There was a lot of tuning needed on the content side to ensure muzzle flashes and skyboxes were in the proper luminosity band for example. Also, there is a lot of tuning from myself, working with the art director and engineers, on exposed parameters to ensure the HDR looks proper. Sometimes you can tune it to look "great" for one level but then the next level looks awful with those tuning values, so you need to be very careful to have global HDR values setup that still look great but also can handle a wide range of scenarios and maintain the artist’s intent.
Finally, we spent a good amount of time ensuring we had a robust calibration screen that represented a range of lighting scenarios. We implemented the ability to hide the calibration image while tuning the HDR sliders and also exposed a curve to represent your HDR output. Finally, adding controls like an HDR contrast and max brightness really helped empower the player to get the look they wanted.
Gears 5 features top-notch animations coupled with beautiful real-time cutscenes. Can you elaborate how you captured the game’s excellent performances?
Penty: We made the switch to FaceWare to process all of our cinematic performances on Gears 5, which really helped with the fidelity. We, of course, have our talented team of animators that work on top of that data to bring the performance to life.
Our facial animation is primarily blend shape based, with joints used as "tuners" to allow us to modify the blendshapes. We went back and cleaned up all of our facial shapes to improve the facial animation quality as well as re-built our compression and expansion normal maps for the face from Gears of War 4. We then drive the blending of the expansion and compression normal maps as well as blood flow to the face with 19 different wrinkle regions. Bringing all these improvements together combined with the character shader improvements really brought the characters to life.
The environments in the game are more detailed and diverse than ever before. Can you speak to how you designed them?
Penty: From a technical point of view, it was really important for us to have a tight pixel density given we knew from the beginning this was going to be an Xbox One X product, so we added a detail texture pass to most objects in the game. We also put in a lot of effort to speeding up our environment Materials on the GPU so we could push more object density in Gears 5 despite it being a 60 fps game. Finally, we combined this with a HLOD pass to allow us more draw calls.
Hanbeck: Gears 5 features the most diverse set of environments ever in a Gears game. As such, it was important to us to make sure that each Biome had a distinct visual signature and left a lasting impression. A lot of attention was paid to this from an art direction standpoint, making sure we pushed the colors, tone and atmosphere as far as we could.
Considering Gears 5 introduces open-world segments, a first for the series, can you talk about what it was like implementing those sections?
Hanbeck: The larger spaces in the game that feature open-world elements posed a significant challenge on the art side. There was a very tight balancing act between performance and being able to put enough objects in the world to make sure it felt alive. At the end of the day, we had to come to grips with the fact that we would not be able to have the same density we were used to from our more linear levels. The bottom line is that you really don’t need as much density when you are going through the world at 10 times the speed you normally would when travelling on the Skiff. I think we found the right level of detail to make the world look not empty and be somewhat believable.
Searcy: The whole idea of including more open, exploration areas was to evolve the "palate cleanser" vehicle sections from past games to include more player choice. The first step was to build the Skiff, a new wind-based vehicle. It was key that it was fun to drive on its own. After that, we went through many iterations of the open areas trying to get the right distance, speed, and shape for exploration. The last piece of the puzzle was actually Jack's progression. Once we had the loop of finding parts for Jack, we were able to build pockets of interesting combat that rewarded you with elements that affected the core game loop, allowing players to explore and pace out their experience between story missions however they wanted.
Penty: Overall, I'm really happy with the visual quality bar we were able to hit while achieving 60 fps in the overworld areas. It was a tight collaboration between, art, design, and tech to ensure the Xbox stayed in memory and had minimal pop-in. We leveraged the HLOD systems very heavily for the open-world segments due to the sight lines, as well as Unreal's landscape system. Artists would generate the landscape height and texture maps initially in World Machine then import those into the landscape system in Unreal and continue to sculpt from there. All the landscape shaders ran tessellation shaders for the highest possible geometry density. The open-world areas were also a big incentive for us to move away from baked shadow maps and move to real-time cascades and ray-traced distance field shadows, as we really didn't want to store shadows maps for these huge spaces.
Gears 5 features impressive destruction set pieces that include destructible cover, and breakable ice. How did the studio implement that into the game?
Penty: Similar to our particle investigations, we found that destructible cover had a really intense CPU overhead on Gears of War 4 and we needed to address that to go to 60 fps on Xbox One X. We extended our Swift Particle System into a Swift Destruction System, which was essentially a vertex offset based system built using the Material Editor. We tied Swift Destruction tightly to our Houdini Engine fracture tool, which would bake out all the data needed for the simulation into the UV sets (example: fracture points). For our shattering ice grounds, we had a vertex shader setup for that effect that would recognize if the ice had been just cracked or completed shattered. We would then have to work with design to setup a Blueprint to properly update the player collision when the ice had hit the damage threshold. We also used Alembic Cache simulation to drive vertex streamed destruction that was created offline in Houdini. This was great for set-piece destruction moments outside of the play-space.
Gears 5 introduces new inventive enemies and gruesome bosses. How did you approach designing them?
Searcy: Our enemy process is something we really worked on in Gears of War 4. We start with the role we want the enemy to have against the player. Should they flush the player? Pin them? Do they alternate through roles? Once we prove out the core effects it will have on the player in prototypes, we start to layer it with other enemies to see how it can be used in different combat encounters. With the design requirements in mind, Concept Art starts exploring the look and silhouette. Eventually a rigged proxy model is added to the functional prototype and we put the enemy through its paces in Campaign and Horde encounters to iterate on its behaviors and shape before we finish things like audio, VFX, and animation. One of the great things about working on Gears is that each enemy has a specific combat role and presence on the battlefield. Getting to conceive of new monsters and layer them into Gears is one of the best parts of the design process.
The multiplayer maps are varied in size and offer fantastic layouts. How did the team approach designing these stages?
Multiplayer Design Director Ryan Cleven: In Gears of War 4, we wanted to make maps that felt like they belonged comfortably within the legacy of the previous Gears games. In Gears 5, we wanted to push the boundaries of what it meant to be a Gears of War map. We wanted to push further out of our comfort zone and try things that we hadn't done before. We broke the axis of symmetry on a few maps, making them rotationally similar instead of simply symmetric. We added a lot more dynamic elements that have significant gameplay impact. We pushed verticality further than ever before. We tried more organic shapes with terrain meshes that don't normally work in the Gears cover system.
Beyond the regular 5v5 maps, we added a completely new system with our modular tile system. We modified the game to load a recipe of tiles to create a layout for PvE or PvP modes. We used procedural systems in combination with a map builder made specifically for Gears 5 to make our new Escape mode.
Considering Gears 5 features new AI, can you talk about what has been improved upon here since Gears of War 4?
Studio Technical Director Mike Rayner: We added support for flying creatures and volume-based pathfinding, which was used to good effect by the new swarm flock enemies in Gears 5.
Player-initiated combat was a new addition to Gears 5 and to support that we enhanced our sensing systems [to] support a more robust idle state with idle behaviors, and introduced a new alert sensing state with dead body detection, investigating noises, and more.
We also added a number of new AI systems to support our game's needs:
- Influence maps (used when pathfinding and searching for cover)
- We replaced the Environment Query System (EQS) by a line of sight caching scheme (similar to our cover-based LOS caches), which allowed non-cover based enemies to efficiently use the space around them.
- Smart AI Objects for idle interactions with the world (replaces many scripted moments)
- A Tactic system to allow for coordination between characters
- Dynamic combat zone tactic to distribute enemies around larger combat spaces
- Flanking tactic, Banzai tactic
Navigation Tools and Systems - we provided an expanded tool set for designers that included:
- Moving navmesh platforms with navigation between the platforms
- Rotating and stitching navigation meshes and volumes to support our procedural game modes (e.g., Escape)
- Dynamic open-world navigation mesh building
Gears 5 is a visual showcase that not only looks great but runs well across a variety of platforms. How did the studio optimize the game so well?
Rayner: From the outset of the project, we targeted 60 fps against a fairly low-spec CPU (Xbox One and equivalent min-spec PC CPU). We knew the CPU was going to be our primary bottleneck, so we invested to keep our simulation as lean as possible. We took advantage of UE4's multi-core parallel rendering, parallel animation and physics support along with our own custom multi-core optimizations to leverage multiple cores and scale performance across a wide range of CPUs. We also invested in some custom simulation LOD management, which would dynamically adjust local client simulation fidelity of characters and VFX to fit a fixed time-budget. While we embraced Blueprints for prototyping, we carefully monitored performance and moved heavier systems and common patterns into C++ code to keep the simulation lean.
For engine performance and visual settings, we leveraged Unreal's scalability options, properties, and console variables to good effect to ensure the game is optimally tuned for each platform. We defined clear budgets for content which we validate on development PCs. This along with UE4's Static Mesh LOD and HLOD systems, and using dynamic temporal upscaling allows us to scale our content to look good and perform across a wide range of PC hardware from min and recommended to ultra spec PCs as well as consoles [like] Xbox One S up to Xbox One X.
We chose a reasonable minimum spec for PC and console to ensure memory and performance was kept in-check across all devices, we validated this through automated testing and telemetry from play-sessions.
Penty: We daily tracked our percent at 60 fps for all of our levels in our QA playthroughs (example: 85 percent at 60 fps). Our performance reports would also include stats like "Bound by Render Thread: five percent of the time," so we knew what threads would be holding up performance. This allowed our technical art and engineering teams to jump into a level with an idea as to what had to be brought down (example: Render Thread bound might imply we have too many shadow casters or too many objects drawing). While we were focusing on the content optimizations daily, I would usually be tuning the global scalability settings per platform/per game mode/per split screen setup, massaging the performance profile.
How important was it to have access to Unreal Engine’s source code for the development of the game?
Rayner: Having access to the source code was crucial to the optimization, debugging, and agility of AAA game development. The ability to diagnose game issues though engine source level debugging is critical to solving subtle issues in our game's implementation. Source access allows us to back-port bug fixes and improvements from Epic without requiring a full-engine upgrade and allows us to fix and make improvements to the engine directly (often contributing these changes back to Epic). Reading source code, and engine developer comments enables our team to understand the engine at the same level that Epic does, which gives us confidence we are using it effectively, and when absolutely necessary a pathway to extend and evolve the engine directly to meet our needs. We rebuild the engine from source ourselves, which enables us to work with newer versions of compilers, and leverage runtime/game specific compiler and linker optimization like performance guided optimization.
Gears 5 features both three-player local and online co-op. Was this a technical challenge to implement?
Rayner: Unreal Engine supports multiple viewports and has split-screen support, so a lot of the low-level support is in place. We invested in input/focus management to support multiple user sign-in to ensure this was a good user experience. Performance is always a challenge with multiple view ports. On Xbox One S, we needed to hold the same fps in split-screen as we did in full-screen models. We approached this by defining different scalability settings and visual options for split screen modes, as well as by introducing some detail tags, which allowed us to turn off high detail objects on lower-spec hardware. For Xbox One X, we opted to drop from 60 fps (full screen) to 30 fps (split-screen). This allowed us to retain the highest quality visual settings and native resolutions in split-screen modes, which we think was a good trade off.
Online co-op is also something UE4 supports with minimal effort provided you have implemented networking replication thoughtfully. For co-op, one of the up to three players is a client hosted listen-server, while the others are clients. Thought and effort is required to ensure co-op experiences were replicated correctly, the best approach here is to ensure level designers and engineers are testing standalone co-op to ensure everything is replicating and working as expected as part of the local review process before submitting changes. We had technical level designers and engineers that worked with our level designers to assist with replication (often handled in C++) and to create common reusable systems that abstract away replication and support save/load and checkpoints.
Allowing players to join games in progress over the network also presents its own challenges, particularly since we have few restrictions around when a player can join. They can join as the game is loading, in the middle of a fire-fight, while we’re playing cinematics or transitioning levels, which presents a number of challenges. For example, keeping the experience as seamless as possible, synchronizing our real-time cinematics and transitioning smoothly into gameplay for multiple players is difficult on its own. Adding in join-in-progress and bad networking conditions greatly increases the complexity. We avoid showing loading screens as much as possible, but had to keep them as a stop-gap for the most difficult moments. The key is to have technical content creators in cinematics, animation, and design that can focus on fixing and mitigating those edge cases, while having engineers create a system that is strong and flexible for the many ways that content comes together.
Considering the game introduces a new multiplayer Escape mode that forces players to escape from the Hive, can you talk about what inspired this mode?
Cleven: As fans of Horde mode, we really wanted to create a shorter co-op experience that we could play when we didn't have a full squad together. The original idea was to create a photo-negative of Horde that could stand next to it. Horde is about making a base where the enemies come to you. Escape is about destroying an enemy hive, i.e. You go to the enemy. It still had to be a survival mode just like Horde. The main objective had to still be “survive,” but the twist is you had to find the exit in a maze or dungeon.
The concept drove the need for a new fiction wrapping. We wanted something that got right to the point, with ultra-high stakes that would make sense in the savage world of Sara. The characters still needed to be highly trained like the rest of the Gears, but they needed a different circumstance. Starting with no ammo, with a bomb strapped to their chest, voluntarily getting snatched and waking up deep underground at the heart of a Swarm hive sounded bad ass. It was crazy enough that it felt like it could sit next to a mode where you destroy hundreds of enemies with thousands of bullets.
Was it hard to implement cross-platform support across PC and Xbox One? Rayner: On the whole, cross-platform development with UE4 is fairly straightforward; provided you follow best practices and stay on top of min-spec memory and baseline performance, everything just works.
Unreal Engine is designed to work cross-platform out of the box and is by and large a device/platform agnostic game engine. For Gears 5, we targeted Xbox One, PC (Steam and Windows Store) and dedicated servers all from the same cross-platform code base. You do have to stay on top of warnings and errors generated by the editor and compiler, which will help ensure your content and code will work across all platforms. Beyond that, if it works in editor, it will generally run without issues on other platforms.
As a cross-platform networked game engine supporting variable time-steps (Delta Time), it is worth noting that cross-platform multiplayer worked out of the box. We did make changes to add Xbox Live cross-platform match-matching and worked on few load-time and extreme framerate scenarios differences causing some timing issues between low and highest spec hardware. In the end, cross-platform development with UE4 is one of its key strengths.
How important was it for the team to implement robust settings options for the PC version of the game?
Rayner: With UE4, we are able to detect a player’s hardware and choose appropriate defaults across a wide range of features and scalability settings to ensure players have an excellent experience out of the box no matter what hardware they are running on.
We believe PC players expect to be able to tune a game to their preferences, which could be ultra-high frame rates, extreme visuals, or any custom combination based on their preference. We have over 35 settings that users can adjust. We believe in helping users understand what they are changing so each setting comes with a detailed description of what the setting does as well as the impact it will have on CPU, GPU and VRAM. In addition, settings that are directly visual come with a preview image for each quality level so users know exactly what they are changing.
We also have a variety of controls to manage resolution and frame rate. All of the expected controls are there, including resolution, VSync, and field of view. In addition to these, we've added ways to manage the maximum frame rate of gameplay and cinematics independently so users can play at an unlimited framerate and then enjoy the cinematics (which are all real time) at a lower frame rate but with higher detail. We've also added a minimum frame rate option that will dynamically keep the game running at a specified frame rate (or greater).
Penty: Overall, Gears 5 is quite a scalable game, especially compared to Gears of War 4. So it scales up beautifully to high-end PCs, or can scale down to low-end hardware. I'm especially a fan of our Ultra spec shadows, volume fog, and reflections, which look amazing.
Did the development team have any favorite Unreal Engine tools or features?
Penty: It is hard to pick a single feature, but for me personally, I'd probably have to go with the temporal upscaling as my favorite feature. It allowed us to really push the visual quality and not worry about the frame getting soft with resolution scaling in-play. My close second would be the Volumetric Fog system and the level of atmosphere it allowed us to add to the game.
Mike Rayner: The Replay System formed the basis for which we implemented our Kill Cam feature for dedicated multiplayer servers. The Replay system is a powerful addition to the engine, and through it we were able to add this long requested contemporary multiplayer feature to Gears 5 with far less effort than if we were to build the feature from the ground up.
For more information on Gears 5, visit: