There was an outage of our database servers that support our forums today. While we were able to recover from the outage, unfortunately there was a data loss suffered.
What happened:
- During the process of removing an unused database cluster, we, apparently, accidentally performed the operation on the live environment supporting the forums.
- We immediately realized the mistake, and began the recovery from our backup. However, the snapshot data was also removed so we could not fully recover the data.
The Impact:
- Forum outage affected all users access to forums.unrealengine.com from 2:58PM EDT to 5:43PM EDT
- Forum data was lost for any posts after approximately 8:00 PM EDT yesterday.
Future Prevention:
- We are currently reviewing our production procedures and data recovery strategy and will provide an update as soon as a full review is complete.
We apologize for the outage and loss of data, and the inconvenience this caused. I will update this forum post with more detailed information about how this happened and the preventative measures we will take to prevent it from happening again.