5.2.2017

Tracking Down GPU Hangs with NVIDIA Aftermath and 4.15.2

By Marcus Wassmer

Unreal Engine’s crash reporter gives us a ton of valuable information when a user crashes, most importantly where in the source code the crash occurred. We rely on this data to track down and fix widespread issues in an effort to improve overall stability.

Tracking down GPU crashes is much more difficult. Until now, the graphics driver has not been able to tell us what the graphics card was doing when the crash occurred. We’re missing that context that is so valuable when tracking down CPU crashes: where in the source code the crash occurred. Sometimes we can ask users to enable the Direct3D debug layer (-d3ddebug), as it catches some bugs that may result in a GPU crash but it cannot catch everything.

NVIDIA Aftermath provides some of this crucial information about what the GPU was doing when it crashed. We don’t yet get an exact location in the source code but we do know where during the frame the crash occurred, for example in a particular postprocessing pass.

Where you come in

Sometime around 4.14 Preview 1 (October), we began to see a lot of reports come across regarding a specific D3D crash but have not had the appropriate information to act upon. We have been tracking this issue with the ticket UE-42280.

It has been frustrating for us to see these reports come across for the last few months without being able to fix the core issues, so we’ve been working closely with NVIDIA to better understand what is happening. Recently, NVIDIA developed a new technology called Aftermath that gives us additional information to assist in tracking down the root of these crashes.

UnrealEngine%2Fblog%2Fhelping-us-track-down-gpu-hangs-with-nvidia-aftermath-and-4-15-2%2FAftermath_Logging-734x163-2a27ee017b7f80fea937a3d5fe08483501815f93

Starting with 4.15.2, the engine will ship with NVIDIA Aftermath integrated, giving us the appropriate info to track down what’s happening with GPU hangs. Your crash reports are an immense help when it comes to fixing bugs in the engine, and we encourage you to continue sending them along. When submitting your crash report, please also make sure to enable the checkbox to include your log files. If you’re currently working with a version of 4.15, we’d really appreciate it if you upgraded to 4.15.2 and update your NVIDIA drivers to 381.89 so we can act fast on getting these bugs squashed.

For more information about NVIDIA Aftermath, please visit https://developer.nvidia.com/nvidia-aftermath.

Recent Posts

Datasmith and Unreal Engine Drive Real-time Architectural Design at Neoscape

Applying a real-time approach to strategy and branding in the real estate w...

Unreal Engine 4.19 Preview 1 Now Available

The first preview for Unreal Engine 4.19 is now available for download o

...

Epic Games Announces New Round of Unreal Dev Grants Totaling over $200,000

Chosen developers around the globe building independent games, tools and an...