October 29, 2014

Debugging and Optimizing Memory

By Ben Zeigler

When a game reaches a certain stage of development, it becomes critical to figure out what exactly it’s loading into memory and why.  As new assets are built, games tend to become larger and larger until load times slow to a crawl and the game starts to run out of memory. Luckily, UE4 has some useful tools built in to track down what’s in memory and why. I use these tools and techniques here at Epic to optimize Fortnite’s memory use and load times.

Looking for problems with Memreport

My first step is always to use the memreport command. To use it, open the console with ` and run "memreport" for a fast report or "memreport -full" for a more complete report. Then, look inside YourGame/Saved/Profiling/MemReports for .memreport files tagged with the map name and time stamp. These are simple text files that contain the output of running all commands inside the [MemReportCommands] (and [MemReportFullCommands] for -full) section of BaseEngine.ini. You can change what commands run for your game if you wish in your own engine.ini file, but the default commands are a good place to start.

The first part of the memreport file lists the overall memory usage, with sections for platform-specific memory usage and all registered memory stats. You can search the source for the name of a memory stat like "STAT_PixelShaderMemory" to see exactly what contributes to it. The next section is the output of the "obj list -alphasort" command, which lists all UObject classes and how much memory they use. After that are some sections for rendering memory, the status of loaded streaming levels, and spawned actors. The -full command also has sections for individual assets like static meshes or textures and is useful for looking for individual expensive assets.

The output of the "obj list" needs a bit of explaining. Here are a few lines from a Fortnite report:

























Using memreport you can quickly see where your memory is going. It's also very useful to take before and after memreports, and use a text diff tool to compare them. They're sorted so they diff properly. Let's use an example from Fortnite to show how you would use tools in the engine to figure out why a certain asset is loaded. I first looked for issues by searching the output of "memreport -full" and looking for large assets that seem out of place. I found a version of this line in a memreport:The first column is the class name followed by the number of instances of that class. For the memory columns, you care about the first and last columns, which are the accumulated memory of all instances of that class. NumKBytes is the amount of memory used by the UObject's body in memory, while ExclusiveResKBytes is the amount of memory used by non-UObject "resources" that are solely owned by that UObject, such as sound buffers. UObject::GetResourceSize is the function that determines the resource size for an object. ResKBytes is less useful in this case because it includes shared resources, so in the Material case, the ResKBytes total is artificially inflated by including the same shared textures once per Material. So, to see the amount of memory used by a certain class, take the 1st and 4th column values and add them.

StaticMesh .../S_Hex_Urban_Standard_03.S_Hex_Urban_Standard_03  1K 1K 14306K 3220K

We have a single static mesh that is taking a full 3MB of memory, and it's not a mesh that I expected to be loaded because it's not needed by the part of the game I was playing at the time. Something is referencing this asset and loading it in when it doesn't need to. I need to figure out why.

There are two good techniques to track down errant references. The first is to load up the editor and find the asset in question in the content browser. Then, right click it and select Reference Viewer, which will give you a view like this:

Reference Viewer

With the reference viewer you can quickly navigate references and see what is referring to an object. In this case a blueprint is referring to it, and if you double click that blueprint, you can then see what refers to THAT. Using this, I can quickly gather a few suspects for what might be loading the expensive mesh.

The next tool to use is the obj refs command. From within the instance of the game where I took the profile, I run "obj refs name= S_Hex_Urban_Standard_03 shortest" from the console. This takes a few seconds and then outputs to the log a dump of reference chains from the GC root to the target object. Here's a sample of the output for this example:

(root) World /Game/Maps/Zones/Zone_Temperate_Urban.TheWorld->CurrentLevel

Level .../Zone_Temperate_Urban.TheWorld:PersistentLevel->ULevel::AddReferencedObjects() 

FortWorldManager .../PersistentLevel.FortWorldManager_0->CurrentWorldRecord

FortWorldRecord .../PersistentLevel.FortWorldManager_0.FortWorldRecord_1->ZoneTheme

(standalone) FortZoneTheme .../ZoneTheme_Urban.ZoneTheme_Urban->HexTileClass

BlueprintGeneratedClass .../HexTile_Urban01.HexTile_Urban01_C->UClass::AddReferencedObjects()

HexTile_Urban01_C .../HexTile_Urban01.Default__HexTile_Urban01_C->Hex Deco Meshes

(target) StaticMesh .../S_Hex_Urban_Standard_03.S_Hex_Urban_Standard_0

This output is a chain of references from a root object that is never GC’d to the targeted object. I shortened the paths to save space. For each line of the output it starts with an optional note like (root), then the object's class, then the object's full path, and finally what aspect of that object holds the reference. In this example some of the references are held by custom AddReferencedObjects() functions, while others are held by editable UObject properties.

Reading that output from top to bottom, I can see that the blueprint is referenced by a FortZoneTheme, which is in turn referenced by part of the save game system in Fortnite. In this specific case I plan on changing the HexTileClass property in FortZoneTheme from a TSubclassOf<> to a TAssetSubclassOf<>, which will stop the referenced blueprint from getting loaded until I explicitly load it myself.

This is just one possible solution, but in my experience, stopping your game from loading unneeded assets is the easiest and most effective method to reduce memory and improve load times. If you have more questions head over to the forums, and feel free to comment below.