April 12, 2018

AI in Unreal Engine: learning through virtual simulations

By Sebastien Miglio

Now far more than just a favorite go-to theme for science fiction, Artificial Intelligence is here, and it’s poised to change our lives in many revolutionary, exciting ways in the years to come. Advancements in self-driving cars, adaptive robotics, and other AI-driven technologies are just the beginning of what’s to come as we step closer to fully realizing many once-futuristic concepts in today’s modern world.

Developing and teaching the artificial brains required to fully realize and implement these wild innovations can be a complex, time-consuming process, but Unreal Engine is playing a major role in creating forward progress to help train and validate AI for real-world applications.

Training self-driving car AI in Unreal Engine

Given all the buzz surrounding it, the evolution and inevitable implementation of self-driving cars on our roadways is a delicate if not utterly fascinating subject. Public safety is clearly a prime concern for putting autonomous vehicles into play, which means the meticulous and thorough process of training the AI is a task handled with care. It requires lots of flexibility and room to iterate to ensure the algorithms are given the sharpest understanding for maximum performance.

For many startups making leaps and bounds in this space, virtual simulation is an essential ingredient for preparing these adaptive algorithms and complex machines for real-world road testing. Unreal Engine is a perfect fit for helping to tune and perfect autonomous vehicle AI in a photorealistic virtual space, because it makes it possible to very quickly recreate and adjust an infinite range of possible test conditions—including lighting, weather, and beyond. Beyond that, having all of the simulations done in a deterministic environment that allows identical re-simulation across numerous tests is vital for measuring improvement and performance consistently over time.

Self-driving car AI benefits from virtual training using Unreal Engine across millions of virtual miles in a simulated space.

Using Unreal Engine to build its next-generation simulation pipeline, Silicon Valley robotics and self-driving car startup Zoox captures real-world city street data and rebuilds everything virtually inside Unreal to provide a baseline “ground truth” for improving its AI algorithm technology. Instead of manually rotoscoping video footage to teach the AI what’s what, the developers can use their foundational street data paired with our real-time engine to easily let the AI differentiate between buildings, pedestrians, curbs, traffic, and more in a much faster way.

Virtual simulation is a powerful technique for amplifying and enhancing the training process, considering the average ratio of physical miles versus virtual miles driven by AI is about 1:1,000. This lets AI clock billions of virtual miles in a simulation driving environment before hitting the streets for the real deal.

The New AI Proving Grounds

Cars aren’t the only autonomous vehicles being tested in the AI research space, especially as more companies become intrigued with the potential of drone technology. Microsoft AirSim is an open-source plugin for Unreal Engine that provides a real-time platform for developing and assisting simulation for testing and AI training, and it can be used for drones, automobiles, and other vehicles.

Photorealistic virtual worlds offer a perfect testing ground for AI research in an environment that can simulate real-world conditions that prove difficult to test in otherwise.

Testing drones and other autonomous vehicles in contained environments has its limitation, which is why creating a safe, dynamic photorealistic environment in the virtual world can give AI research far more control over exploring how their algorithms respond and adapt to conditions that can be hard to replicate normally. Having control over all facets of a simulation, for example, lets researchers test what the effect of different wind speeds might have on an AI-driven virtual drone’s performance.

Microsoft’s AirSim plug-in for Unreal Engine empowers AI researchers to train and test their autonomous vehicle algorithms in a safe, dynamic virtual environment.

In a training-driven scenario, ensuring the AI brain is learning based-on correct and accurate data and sensor information is vital to the success and safety of the technology. Microsoft’s goal for the research project is continue adapting AirSim as a tool for AI experimentation, deep learning, and reinforcement learning.

We’ve entered and exciting era for the evolution of Artificial Intelligence where the democratization of tools and technologies will soon make the impossible possible. With Unreal Engine powering AI development and training approaches across many different design and research industries, there’s a tremendous level of potential progress that can be unlocked in the advancement of artificial intelligence for future generations.

Ready to join the wave of innovators changing the world through real-time technology? Try Unreal Engine today for free!