Image courtesy of Scania CV AB / GEISTT AB

Real-time simulation of new HMI concepts at Scania

November 6, 2020
GEISTT AB is a highly specialized agency that provides consultancy services, practical methods and tools, and applied R&D that enhances overall human-system performance and safety. GEISTT uses an applied, multi-disciplinary, and research-oriented approach based on practical scientific expertise in contextualized human cognition and behavior, combined with a deep understanding of advanced technology and organizational effectiveness.

The firm has been working with the interaction design team at heavy-vehicle manufacturer Scania since 2013, providing R&D services, concept development, and simulation-based testing. These simulations are used by Scania to analyze effects of human-machine interface (HMI) concepts for everything from new UI features to procedures for interactions with autonomous vehicles. 

To allow for more resource-effective and immersive testing of future concepts earlier in the development process, GEISTT moved Scania’s human-in-the-loop simulation capabilities from a resource-consuming purpose-built simulator to Unreal Engine. 
Image courtesy of Scania CV AB / GEISTT AB
The business value has been immeasurable. “In the end, it comes to quantifying how much an effective simulation capability is worth for the company,” says Jon Friström, UX Researcher & Cognitive Design Engineer at GEISTT, consulting as Product Owner for Scania’s HMI simulation team. “What is a better product worth for the company? How much money is saved by detecting problems with the user experience early in the development process - and what would the alternative cost be?”

Immersive HMI testing with real-time simulation

Scania’s interaction design development is often iterative. Each iteration pushes the concepts that are developed closer to a final product. Tests of the concepts are carried out to provide feedback on concept performance. The goal is almost always to understand how the concept would perform in its intended natural context before it is deployed or proposed for further development. This gives the development team a chance to correct minor issues at an early stage, or even rethink the entire concept. Problems found earlier are always easier and less expensive to fix.

As the development projects progress, the concept testing iteration increases in realism, complexity, and cost. At Scania’s interaction design group, early tests are commonly accomplished using paper prototypes, while later tests are more resource-demanding and performed in a driving simulator. Prior to its use of modern real-time technologies, a typical development project at Scania would consist of a few early user tests with wireframes on paper, perhaps an interactive prototype on a computer, then at the end, a simulator study for validation of the concept.

This meant that a novel concept was only tested once in a simulator environment, and the designers and developers could not test their concept again after receiving feedback from the simulator study. Now, using Unreal Engine, the team can test concepts in immersive simulations earlier in the development process. The engine’s strong connection to C++ and its open-source nature enables the team to easily adapt the technology for specific use cases. “For instance, it’s relatively easy for us to use Unreal Engine as a platform to connect different plugins and systems in a very flexible manner,” explains Friström. 

Driver interaction simulation

One of the major areas the Scania interaction design team focuses on is driver interaction simulation. This entails testing HMI concepts as part of a system that includes the driver and its vehicle, and often other cars or pedestrians.
Image courtesy of Scania CV AB / GEISTT AB
In testing prototypes of advanced driver-assistance systems (ADAS), the team compares variations of the prototype system using actual drivers. In many of these tests, participants are repeatedly and unexpectedly exposed to situations that would result in extreme danger for the driver or persons around the vehicle if the test had been conducted on a real-world test track or an actual road with real traffic.

Some tests involve evaluating advanced features for specific user groups. Timber drivers and inner-city distributors have diverse and specific needs, for example. The team evaluates features that could help these user groups specifically and measures work effectiveness and efficiency gains.

New UI concepts are also explored. These might include side-by-side tests of new UI principles—such as variants of early infotainment system concepts—performed by real drivers to identify the best interaction principles and combine them in the next iteration of the concepts.

In most tests, the simulation team tracks objective data like brake pressure or lane deviation; physiological data such as reaction times, eye tracking, and heart rate variability; and subjective data including pleasurability/UX survey ratings, and feedback on the experience through interviews with testers. 

Human-automation interaction simulation

Scania’s interaction design team also creates simulations to test human-automation interaction. These can be divided into three categories. 

The first explores the relationship between the vehicle automation and the driver inside it. These types of research projects revolve around procedures for giving and taking over control of the vehicle in different types of situations—after a long rest or during an emergency, for example.
Image courtesy of Scania CV AB / GEISTT AB
The tests explore how the driver role varies with a higher degree of vehicle autonomy to determine how to set up routines to prevent automation fatigue and give the driver a correct mental model of the intentions and world perception of the automation. These studies have been conducted both in VR and in driving simulators.

The second type of human-automation interaction tests focus on the research and development of a control room environment to manage various types of fleets of autonomous vehicles, such as special vehicles in an open pit mine or buses as part of the public transportation system. 

Finally, the team explores the control and the perception of the intentions of autonomous vehicles, as an external actor. This entails research into how to remotely control automated vehicles—for example, self-driving dump trucks in mines, or research into how to design external human-machine interfaces on the vehicles to better understand its status and intentions. These types of projects use mainly VR to immerse the researcher or test participant into the simulations.
Image courtesy of Scania CV AB / GEISTT AB
Friström notes that research projects like those conducted by Scania’s interaction design team are steadily on the rise. “We see a general increase of projects researching and testing the relationship between humans and highly automated systems in various industries, due to the increasing complexity of the systems, often with higher and higher degrees of artificial intelligence (AI),that interact in different scenarios—so we expect to see an increased use of human-in-the loop real-time simulations in the future,” he says. 

For Friström’s team specifically, adopting Unreal Engine as the real-time platform of choice has had a significant impact on ROI. “Due to the nature of what we are developing, using Unreal Engine compared to other similar real-time engines has definitely been cost-effective,” he says. “We have direct access to the source code without paying extra, the community support is great, and technical support from Epic is well worth its price.”

An open platform for development 

Working on an open platform such as a game engine has considerable advantages for those testing different HMI concepts. “The biggest benefit for us has been the ability to use Unreal Engine as a flexible platform that we can use to integrate several different closed systems,” explains Friström.

The vehicle dynamics in his team’s simulator are handled both by an external closed module—TruckSim by Mechanical Simulation—and the built-in vehicle models provided by Unreal Engine. 
Image courtesy of Scania CV AB / GEISTT AB
Similarly, the vehicle sound in the simulation is handled both by a closed external module—Wwise by Audiokinetic —and by the built-in audio tools in Unreal Engine. Randomly-generated traffic navigates with closed external software Sumo, but the team also uses the engine’s built-in tools to navigate the surrounding traffic under certain circumstances. “The flexibility to shape the real-time engine to accommodate and use these separate modules is a great asset of Unreal Engine,” says Friström.

Another essential feature the team uses is nDisplay, the technology that renders Unreal Engine scenes on multiple synchronized display devices. nDisplay enables GEISTT’s simulators to offload work on many computer nodes in a network instead of only relying on one computer. “By doing so, we can render the environment to an infinite number of projection screens,” explains Friström. 

Meeting the HMI simulation challenges of the future 

Friström and his team have fully embraced the switch to Unreal Engine for their HMI concept testing. “Compared to the simulator platform we used before, UE4 has a broader, more supportive community,” he explains.

This access to knowledge and support has been invaluable in getting GEISTT’s simulator up and running. As for the future, Friström predicts simulations will play an increasingly important role in optimizing user experience for vehicles. “What we can see is that all automotive companies are using different forms of UX simulations to an increasing degree, and this trend is clear in other industries as well.”
Image courtesy of Scania CV AB / GEISTT AB
Using Unreal Engine as a platform for HMI concepting and testing, GEISTT and Scania are well-placed to meet the challenges of the future.

    Let's talk!

    Interested in finding out how you could unleash Unreal Engine’s potential for training and simulation? Get in touch to start that conversation.
    Event
    Unreal Build: Virtual Production 2020
    Coming to you from the comfort of your own home on Tuesday, November 10, Unreal Build: Virtual Production is a free half-day virtual event that showcases incredible projects and innovations from film and television industry trailblazers.
    Event
    News
    Connect with the Unreal Engine community online
    Although many physical events around the world are currently on hold, there are plenty of places to connect with the Unreal Engine community online.
    News
    Interview
    11-person studio Neon Giant delivering next-gen passion project The Ascent
    Comprised of industry veterans who worked on the Wolfenstein, DOOM, and Gears of War franchises, Neon Giant is a small indie studio based in Sweden that’s developing a heavy hitter for Xbox Series X. 
    Interview