June 18, 2014

Today’s Unreal Engine Forums Outage

By Chris Gerhardt

There was an outage of our database servers that support our forums today. While we were able to recover from the outage, unfortunately there was a data loss suffered. 

What happened:

  • During the process of removing an unused database cluster, we, apparently, accidentally performed the operation on the live environment supporting the forums. 
  • We immediately realized the mistake, and began the recovery from our backup. However, the snapshot data was also removed so we could not fully recover the data.

The Impact:

  • Forum outage affected all users access to forums.unrealengine.com from 2:58PM EDT to 5:43PM EDT 
  • Forum data was lost for any posts after approximately 8:00 PM EDT yesterday.

Future Prevention:

  • We are currently reviewing our production procedures and data recovery strategy and will provide an update as soon as a full review is complete.

We apologize for the outage and loss of data, and the inconvenience this caused. I will update this forum post with more detailed information about how this happened and the preventative measures we will take to prevent it from happening again.