August 25, 2017

Getting Started with UE4 and ARKit

By Nick Whiting

On June 5th, Apple, with its revolutionary ARKit, showed us how you can tell your own stories in a whole new way with iOS devices. As we continue to integrate ARKit into Unreal Engine, we're inspired by the Wingnut AR demo at WWDC, and we’re truly excited about the AR experiences so many Unreal developers are already creating for iOS devices.

Will you use ARKit for architectural design, to make a game that changes how players interact with their environment, or something entirely different?

Bring your ideas to life with ARKit using this Quick Start Guide for Unreal Engine 4.17!

Unreal+Engine%2FblogAssets%2F2017%2FAUGUST+2017%2FARKit+Sample+Released%2F770_ARKitSampleReleased-770x404-fdc202e4d3fe46fce114a597fe900460c23d5ae9

Project Setup

To use a project with the experimental Apple ARKit plugin, you must first have the following prerequisites:

  • Unreal Engine 4.17, with ARKit Support
  • XCode 9 Beta 3 or 4
  • iPhone or iPad with iOS 11 Beta

Both XCode 9 Beta 3 or 4 and iPhone or iPad with iOS 11 Beta can be downloaded from http://developer.apple.com.

To ensure that the correct version of XCode is used, run the following command from a terminal window:

        sudo xcode-select -s /Applications/Xcode-beta.app/Contents/Developer

Now, open Unreal Engine 4.17 and create a new project targeted for mobile. Open the project and enable the experimental Apple ARKit plugin from the plugins menu. You may need to build the plugin in order to enable support if you’re working from GitHub source, if prompted.

If you haven’t already, configure the certificates for your device in order to make a signed application by following the instructions in our documentation

Enable the ARKit plugin under Edit > Plugins and restart the editor.

Once the ARKit plugin is enabled, go to Project Settings > iOS. You’ll need to set the following in order to use ARKit:

  • Minimum iOS Version: 11 Beta
  • Additional Plist Data:

<key>NSCameraUsageDescription</key>\n<string>UE4 needs permission to use the camera in order to use ARKit.</string>\n

  • Max Metal Shader Standard to Target: Metal v1.2

Using ARKit in Your Project

There are three major areas of an ARKit project: the camera, the pass-through, and plane detection. We’ll give you a quick intro to each of them to help you get started!

CAMERA SETUP

Fortunately, the camera behaves very similar to our VR camera setup, and should be familiar to those that have set that up. All we need is a new Pawn class that has a special Apple ARKit Camera Component in it. When we run on the device, that Camera Component will move around the virtual world in the same way you’re moving around the real world.

Unreal+Engine%2FblogAssets%2F2017%2FAUGUST+2017%2FARKit+Sample+Released%2FARKitSample_Pic1-770x208-9f5bf132bba2cb6f253fd50b740620a6ed92e626

PASS-THROUGH SETUP

The pass-through lets us take the feed from the camera on the back of your iPhone or iPad, and then use that as a background for your virtual content. This gives the illusion of your content “augmenting” the real world. Here are a few steps that outline the basics of setting up the pass through camera. For a more complete example, refer to the Material setup in the ARSample project linked at the end of this blog.

  • Add an ARKit Texture to your project.

Unreal+Engine%2FblogAssets%2F2017%2FAUGUST+2017%2FARKit+Sample+Released%2FARKitSample_Pic2-770x314-45be1359ad5f70912ff14d39775ec81c9eb9fb72

  • Create a new Material. Change the shading model to Unlit and check the Two-Sided checkbox.

Unreal+Engine%2FblogAssets%2F2017%2FAUGUST+2017%2FARKit+Sample+Released%2FARKitSample_Pic3-770x550-aab37b9ac575fb171355fb4e82cc602cac63f05b

  • Add a Texture Sample node, reference the ARKit Texture you created and plug it into the Emissive channel of the Material. Add a ScreenPosition coordinates node and plug it into the UV input of your ARKit Texture. Note: You’ll want to scale the UVs for the ARKit Texture relative to your screen ratio. For an example of this in action, please refer to the ARSample linked near the end of this blog.

Unreal+Engine%2FblogAssets%2F2017%2FAUGUST+2017%2FARKit+Sample+Released%2FARKitSample_Pic4-770x550-29313731161e40050edee311d58cbeac66015d1a

  • Add a sphere mesh to your scene and scale it up to something that encompasses your content, then apply your newly made Material. This will act as your skybox and will render what the camera sees to the sphere.

PLANE DETECTION

When placing your virtual content in the real world, it’s often useful to find flat surfaces, such as walls, the floor, or table tops. That’s where plane detection comes in. Plane detection lets you ask ARKit to find you flat surfaces around the world and then use those to position your virtual content on. For example, if you were making a virtual cat, plane detection could help you find the floor for it to run around on.

Plane detection happens automatically as you move the device around, and using detected planes with touch input is as simple as calling “Hit Test at Screen Position” function on the ARKit Camera Component. If a plane is found, you’ll be able to access the transform information found in the Apple ARKit Hit Test Result to handle interaction logic with that point in the real world.

Unreal+Engine%2FblogAssets%2F2017%2FAUGUST+2017%2FARKit+Sample+Released%2FARKitSample_Pic5-770x198-72ea15ee182d62a56f2fead96f278ad7f1c986f8

Testing on Device

Once the project is set up, click Launch > (Your iPad or iPhone), which will build the project and deploy it to the device as you would with any other mobile project.

AR Sample Project

Unreal+Engine%2FblogAssets%2F2017%2FAUGUST+2017%2FARKit+Sample+Released%2FARKitSample_Pic6-770x462-d7e8c4d92cd7afd92cb3be38eaf4b5f2ff3a6fe3

To help you get started, we’ve put together a sample project with everything already set up in order to get moving with ARKit and UE4. Download the ARSample project for 4.17.1 here.

Keep in mind that the experimental plugin will change as the technology evolves in these early days. When we officially release the ARKit plugin as part of the engine in 4.18, supporting documentation will be updated to reflect the final architecture.

We can't wait to see what you create with this revolutionary technology, so please share your experiences with ARKit on social media and tag us using @UnrealEngine and #UE4 on TwitterFacebook or Instagram!