Divine Intervention - a behind-the-scenes look at the development of the AR god simulator ARrived

EDITOR'S NOTE: Thanks to the team at Luden.io for taking the time to tell the development story behind ARrived - one of the App Store's Best of 2017. For more information on making AR experiences with Unreal Engine, head here.

September, 6 a.m., San-Francisco, - it's the last chance to enter the iOS 11 launch line before time expires. We must make the deadline, our whole team is sleeping on the other side of the planet, and a "Build Failed" message is facing me from my screen. How did I manage to get into this situation? Well, let me tell you about this.

We’ll Follow the Plan

We were experimenting with AI and drawings identification and decided to make a VR god simulator game. All of us at Luden.io love Black & White to death; we love to draw in VR with our hands and everyone thought that it was a cool idea to control the little men on earth. You draw a rain symbol in the sky — it starts to rain, doesn’t that sound cool?

No Time to Explain

To make the story complete I should mention that we have been working in the VR field for quite a while, but we’ve all agreed that AR is more interesting. In June at WWDC, Apple announced that ARKit would be released in the autumn of 2017, and it took us five minutes to decide that we must jump on that train. What were we doing during those five minutes of decision making? Finding out if Unreal Engine supported ARKit.

Limitations

With the switch from VR to AR already eating up time, and with two months of development left before launch, we decided that we wouldn’t be changing the genre of the game. Some of the development, such as on neural networks, was still useful, but more on this later. Based on the terms and our capabilities, we decided to bet on the updated Sequencer, aiming to make a weekly interactive story about a tribe that lives in the player’s room. We wanted to make the tribe look at the player and react to the camera.

There were a lot of risks.  We did not yet know the rules for what is considered a good tone in AR and it was clear that an AR game session taking longer than 5–10 minutes would be hard on the phone’s battery. At first we thought that showing a large terrain site would be a good idea, but we soon realized that it was necessary to show as little as possible of the real world on the screen in AR. Therefore, we turned to our favorite trick — we created the whole scene from premade Unreal Engine Marketplace assets. We highly recommend this approach, as it allows you to see the whole picture at once, and then  work on the details.

Besides programmers, we have two 3D artists and one game designer working in Blueprints. That’s why we made a system for assembling scenarios and transition logic between them, so that the three team members could work in parallel and embed the scenes in a common system as simply as possible. After three weeks, the base script was implemented and we decided to conduct playtests in order to listen to dozens of complimentary reviews about our ideas, new magic technology and gather general feedback.

The Game vs Reality

When the third player began to check the game out, we realised we were in trouble. We noticed all the players were poking the poor humans about five times per second. They wanted the little lads to react and to be able to zoom in with standard mobile gestures. Our wonderful script and story transitions did not particularly interest anyone. It was like a lightning strike for us, since there was only about a month left.

There was nothing we could do, so we began to rebuild the game into an interactive sandbox inspired by a rather popular game of its time, Pocket God. The good news is we were able to leverage light estimation from the device and use real-world lighting to affect what happened in the game. When it got dark in the room, we sent the tribe to bed and released fireflies into the scene. When we posted this on Twitter, it blew up as everyone liked this combination of AR and the real world.

Rethinking the game's systems so late in the game posed many new design challenges. We tried to keep everything we had made during the past month in the game, including animations, scenes, and transitions. To receive such a hybrid, we added selection buttons to the key moments of the game. All the rest of the management was just pressing on the little guys.

More Tech

The game also lacked wow-factor. We knew how to solve this issue, but we were afraid to do so because of the tight deadlines. However, after we finally gained some courage, we decided to take the risk anyway and added the ability to recognize real-world objects in the game. Just imagine how cool it would be if a virtual tribe reacted to a dog from the real world. Or if the tribe wants to eat, you can show them a banana and they will receive a virtual banana in the game. We implemented the Apple CoreML system to the game using a self-made plugin in Unreal Engine.

Gamescom

We performed playtests as quickly as possible prior to catching a plane to Gamescom 2017. The interactions did their job and it was possible to start polishing the game. The wow-effect from object recognition also worked as smooth as butter. It was a truly new experience for the player. We had to show the game to the journalists with our fingers crossed, in the hope that it would not crash. We were very lucky that everything went smoothly and, our friends from VRFocus even believed the game was already stable, but we knew that was far from the truth.

Besides meeting with journalists at Gamescom, we had one more important task to take care of — speak with the Unreal Engine team. With the release of iOS 11 just prior, Apple had changed the format a bit. While we were able to test the game on our devices, we could not load it into TestFlight, which we needed to begin Apple's review process. After a chat with the Unreal Engine team and some follow-ups, the problem was solved.

PAX

The game became better and better each day, and our hands were not shaking as much when we demonstrated it at PAX. We even had some time to add an option of changing the size of game objects in the same way we zoom photos on iPhone. It was super funny when people discovered this and made the little guys grow to the height of the Washington State Convention Center, where PAX Seattle 2017 was held.

A week after PAX, the launch date for iOS 11 was announced. For us, this meant that the game should be submitted no later than September 15th. We were so nervous that it seemed that we would be able to make it all on time, we even walked through the interfaces and made hints about the possibilities of the game in the form of quests in the upper right corner. Then the player could simply do any crazy things that they wanted to or just perform fun quests like “smash the tree with a lightning”.

In addition, we realized that lighting in different rooms is very diverse, so it’s important to add a quality tracking indicator for the player and give recommendations on how to improve tracking. Otherwise, the player will think that there’s a bug in the game, and that’s why the entire 3D scene is moving somewhere it is not supposed to move to.

Launch

On the day of the launch, we thought everything was good. I was in San Francisco, and the guys from Moscow and Cyprus had created a build. Since they were very tired from this 3-month race, they collected the final build and went to sleep. I saw the sunrise, loaded the build to iTunes and immediately got rejected.

You know, there are some special words in Russian that can help to express the intention to solve a complex problem when it is unclear how to solve it. Using those words, clearing up the cache, and rebuilding the editor I was able to create the necessary build in three hours. iTunes Connect approved it. I think we made the deadline with only a few hours left at that moment.

After the iOS 11 launch, our game was right on the main page of Apple's App Store. We were crazy happy and hyped about this! A few different fun surprises followed later, such as Apple’s award in the end of 2017, but those are stories for another day.

Conclusions

So what conclusions can be made based on this story?

• It is very important to show the quality of the AR tracking and explain how to improve the tracking in AR to the player so that it is more comfortable to play.
• Most players expect AR objects to be interactive.
• The gaming session in AR should be short, about 5 to 10 minutes, because carrying the device for long is not very comfortable and the battery charge will not withstand longer sessions.
• It is better to build a game based on interaction of virtual or real objects than make a massive scene.
• New technologies, such as object recognition and light estimation, can attract players. They like everything interesting and technological.
• Be careful when experimenting with new technologies. Our custom recognition system recognized the journalist’s cat as a dog and as a result received an average rating.
• With Marketplace, you can build a prototype of the project very quickly. Use it.
• Run playtests — the sooner the better. Players perceive the game differently to what we,  developers,  do.
• Communicate with the engine team. They want to help you if they can.
• Make games you love. Otherwise each obstacle on the way to launch will give you an excuse to give up.