September 23, 2019

Streamlining flight operations with VR aircraft simulation

Finnair

Zoan

Spotlight

The management and maintenance of aircraft are complex processes, particularly at major airports. Dozens of planes must be safely coordinated and serviced, involving procedures that have many moving parts. Flight crews must be trained to not only deal with their usual duties, but also to respond to unusual circumstances and emergencies.

Based at Helsinki Airport, the Finnish flag carrier Finnair prides itself on being one of the most forward-thinking operators in this area. The company has long been using cutting-edge machine learning and AI technologies to improve everything from analytics to customer service.

Finnair recently partnered with ZOAN, a virtual reality studio specializing in experience-based services, to pilot a system that simulates the use of AI and visualized data to enable seamless operations. Harnessing the power of Unreal Engine, the studio developed a series of interactive VR and AR experiences based around digital replicas of Helsinki Airport and Finnair aircraft.

Improving situational awareness on the ground

Staff overseeing maintenance and servicing processes need a high degree of situational awareness, which requires familiarity with the airport environment and aircraft as well as a deep understanding of the procedures and their implications.

To aid in situational awareness, ZOAN and Finnair developed a demo of a digital twin system at Helsinki Airport for on-site staff to monitor aspects of operations. Digital twins are virtual replicas that update with live data in real time, giving up-to-the-minute insight into the status of real-world assets and enabling organizations to better manage equipment and inform business decisions.

Simulating the use of various data visualization technologies including AI and VR, Finnair staff could monitor and affect various activities behind the scenes in the demo and thus enable seamless operations. While the digital twin system itself is strictly for demo purposes, the ultimate goal of streamlining operations is to give customers the best possible experience when flying Finnair.

Stepping into the shoes of aircraft maintenance staff

At the Slush 2018 conference, the airline showcased an interactive VR/AR demo that gave visitors a peek into the system, illustrating how Finnair uses it to visualize scenarios, make better judgement calls, and improve customer experience.

“Our interactive VR/AR demo was an excellent tool to present Finnair’s situational awareness initiatives,” says Patrik Etelävuori, Digital Transformation Lead at Finnair. “Visitors could get a comprehensive and realistic idea about the role of data-driven operations for airport staff.”

In the AR simulation, a user can get a taste of running Helsinki hub operations as a supervisor and performing tasks to shorten the turnaround time of aircraft. In the VR experience, they’re part of the ground operations, completing tasks that ground teams perform when readying aircraft for departure.

In creating the demo, the biggest challenge for the team at ZOAN was defining the concept. The experience not only needed to be relevant to Finnair, but also engaging for the public. That meant careful selection of the operational tasks included.

“There were a ton of tasks we could choose from—but we had to think about how to create something fun,” says Miikka Rosendahl, CEO at ZOAN. “We had a list of different experiences and we even prototyped a few, but in the end, we went with the most user-friendly ones.”

With less than two months to deliver, however, the team didn’t have much time to experiment. Though they were working on both projects simultaneously, many of the assets they’d created for the situational awareness demo had to be reworked for the VR demo because of the difference in detail between the two. The situational awareness demo had been built to be viewed from a distance, whereas the assets for the VR experience would be seen up close and required much greater detailing. That effectively meant building a whole new set of virtual replicas for the demo.

“It was the first time we’d done something like this, and we had to build it from the ground up,” Rosendahl explains. The team made use of available 3D libraries, improving the models for real time and making them look like Finnair assets. The team mostly used Blender for the modeling work, exporting OBJ and FBX files into Unreal Engine.

Unreal Engine was the natural choice when it came to creating the real-time VR simulation, Rosendahl says, “because of the tools and integration it provides, plus the fact that it’s the easiest way to achieve high-fidelity visuals and tweak interactions in VR.”

The impressive results were a breakthrough for Finnair, kick-starting its journey into immersive media. The Slush demo provided an asset they could promote internally to show how virtual reality can be used in different areas, and eventually led to a number of other collaborations with ZOAN.

Training cabin crew in a VR environment

One such project had the VR studio diving into the procedures and internal layout of Finnair aircraft to create a practice environment for new cabin crew.
Digital replicas can help reduce employee training costs while simultaneously improving knowledge transfer. Users can make mistakes in the simulation and gain valuable experience—but without real-life consequences.

ZOAN created a VR training simulation proof of concept for the Finnair Flight Academy (FFA) based on the inside of a Finnair A320 aircraft. The goal of the project was to facilitate the education and training of new crew members, allowing them to freely explore the aircraft and learn how to perform key tasks during flight.

The ZOAN team turned the project around in an impressive three months, no mean feat when you’re shooting for inch-to-inch accuracy. “From a functional and technical perspective, coming up with a full plane modeled in two months was a major challenge, considering the details, articulated compartments, and precision of the environment we were aiming for,'' says Rosendahl.

For the 3D models, some ready-made assets were available which ZOAN optimized, including the interior of the cockpit. But the majority were modeled by the team, including smaller, detailed objects inside the plane. They used 3ds Max most of the time, exporting to FBX format.

“We used UE4 for this project because it allows us to go further with the effects, mechanics, and functions inside the experience,” says Rosendahl. “We have a few pre-made functions from previous projects that could be reused for the UE4 projects, which facilitated the process.” The team also used Perforce to sync the project online, enabling a remote programmer in Brazil to support the project.

Leveraging existing assets for future projects

ZOAN was additionally able to create an internal-only video for new staff explaining Finnair’s speed turnaround concept, using some of the 3D models from the digital twin projects it had already worked on. Aimed at further improving training methods, this video neatly summarizes all the different information from the company’s training documents and presentations in a concise way. It provided a further avenue for Finnair to invest in and optimize their training methods, at the same time reducing the expenses of the training system.

Impressed with the visuals and highly detailed environment of the finished project, Finnair is considering the potential to extend this initial proof of concept into a fully functional educational tool. Finnair now has several assets they can use across training and PR, illustrating both the cost efficiency and versatility of real-time rendering for simulated experiences.

To create your own VR simulations, download Unreal Engine today and get started.