Unreal Engine’s crash reporter gives us a ton of valuable information when a user crashes, most importantly where in the source code the crash occurred. We rely on this data to track down and fix widespread issues in an effort to improve overall stability.
Tracking down GPU crashes is much more difficult. Until now, the graphics driver has not been able to tell us what the graphics card was doing when the crash occurred. We’re missing that context that is so valuable when tracking down CPU crashes: where in the source code the crash occurred. Sometimes we can ask users to enable the Direct3D debug layer (-d3ddebug), as it catches some bugs that may result in a GPU crash but it cannot catch everything.
NVIDIA Aftermath provides some of this crucial information about what the GPU was doing when it crashed. We don’t yet get an exact location in the source code but we do know where during the frame the crash occurred, for example in a particular postprocessing pass.
Where you come in
Sometime around 4.14 Preview 1 (October), we began to see a lot of reports come across regarding a specific D3D crash but have not had the appropriate information to act upon. We have been tracking this issue with the ticket UE-42280.
It has been frustrating for us to see these reports come across for the last few months without being able to fix the core issues, so we’ve been working closely with NVIDIA to better understand what is happening. Recently, NVIDIA developed a new technology called Aftermath that gives us additional information to assist in tracking down the root of these crashes.
Starting with 4.15.2, the engine will ship with NVIDIA Aftermath integrated, giving us the appropriate info to track down what’s happening with GPU hangs. Your crash reports are an immense help when it comes to fixing bugs in the engine, and we encourage you to continue sending them along. When submitting your crash report, please also make sure to enable the checkbox to include your log files. If you’re currently working with a version of 4.15, we’d really appreciate it if you upgraded to 4.15.2 and update your NVIDIA drivers to 381.89 so we can act fast on getting these bugs squashed.
For more information about NVIDIA Aftermath, please visit https://developer.nvidia.com/nvidia-aftermath.