Saturday, November 21, 2026
HomeiOS DevelopmentWorking with Picture Monitoring in ARKit

Working with Picture Monitoring in ARKit

[ad_1]

Ranging from iOS 12, builders can use the back-facing digital camera to detect and observe identified photos by utilizing an API referred to as ARImageTrackingConfiguration. What does it imply to you? In case you’ve watched the Harry Potter film earlier than, you most likely bear in mind the wizarding newspaper with animated photos in print. With the picture monitoring configuration, we will flip this right into a actuality by way of an ARKit app.

On this chapter, you’ll learn to use ARImageTrackingConfiguration to carry out picture monitoring and construct an AR newspaper app just like that appeared within the Harry Potter film. It’s not simple to explain how the app works in phrases. So, please head over to this hyperlink and see the demo app in motion. When the app acknowledges a selected picture, it overlays the picture with a video. As you may see within the video, what’s superb is that ARKit tracks the place change of the picture and maintain anchoring the video content material.

Trying fascinating, proper? Let’s get began.

Making a New Venture Utilizing ARKit template

We are going to construct this undertaking from scratch. Assuming you’ve opened Xcode, you can begin by creating a brand new undertaking. Select the Augmented Actuality template and click on Subsequent. For the undertaking data, identify the undertaking ARNewspaper and be sure you set the Content material Know-how to SceneKit.

ARKit-ios-app-template

When you saved the undertaking, Xcode will generate the ARKit undertaking for you. There are a few issues we don’t want within the generated code. By default, the template generates a 3D spaceship, which is loaded when the app launches. Clearly, we don’t want this mannequin. Within the ViewController.swift file, change the next line of code in viewDidLoad() from:

To:

For the viewWillAppear methodology, it’s loaded with ARWorldTrackingConfiguration():

We don’t want it too as a result of we are going to use one other monitoring configuration to deal with the picture monitoring. In the meantime, empty the tactic like this:

Getting ready the Assets

Now that we’ve created the undertaking skeleton, let’s transfer onto the following step and import the required assets. First, obtain the useful resource pack from https://www.appcoda.com/assets/swift5/ARNewspaperResources.zip. Since this app goes to play a video when a specific picture is acknowledged, it’s important to put together the next assets:

  1. A picture to be acknowledged. It may be any picture (JPG/PNG) in your picture library. Or you should utilize the one (i.e. snowboarding.jpg) which comes with our useful resource pack.
  2. A video to be performed after the picture is detected. Once more, it may be any video in mp4 format. Within the useful resource pack, you must discover a video file named snowboarding.mp4.
  3. A doc containing the picture. That is utterly optionally available. However to imitate the Harry Potter newspaper, the useful resource pack features a PDF file named arkit-newspaper.pdf, as proven in determine 45.2. You may print it out in A4 measurement paper for later testing. In case you don’t have a printer, that is okay. You may open the file on an iPad or a Mac. We are going to talk about extra about how one can run the check within the later part.

Subsequent, we are going to import the video file (i.e. snowboarding.mp4) into Xcode. Within the undertaking navigator of Xcode, proper click on ARNewspaper and create a brand new group named Assets. Drag the video file from Finder into Xcode and place it within the Assets folder. When Xcode prompts you to decide on choices for including the file, please be sure you tick the checkbox of ARNewspaper within the Add to targets choice. In case you’ve efficiently imported the video file, you must have the ability to preview it in Xcode.

Setting the Reference Picture for Picture Monitoring

We haven’t imported the picture but. As you recognize, we used so as to add the photographs to the asset catalog. However this time, it wants a bit of additional work for picture monitoring. Open Property and right-click to decide on AR and Textures > New AR Useful resource Group. This creates a brand new useful resource group to specify the reference photos for picture monitoring. You may drag the snowboarding.jpg file (or your individual picture) to the useful resource group.

Whereas we solely have one picture for this demo, you may really put a number of photos for picture monitoring on this group. As you may see, there’s a massive exclamation mark on the picture, which suggests there’s something lacking. Now choose the picture within the group and open the Attributes inspector.

ARKit-image-tracking-reference-image

To detect the precise picture, ARKit requires you to specify the real-world measurement of the picture. Assuming you print the newspaper file on a A4-size paper, the picture ought to have a width of 9cm and a peak of 16cm. So, change the Items to Centimeters and set the width to 9. When you set the width, Xcode routinely calculates the peak worth.

Please be aware that the real-world measurement of the picture varies. In case you print it on a A3-size paper or show the PDF file on a 42-inch display, the bodily measurement of the picture turns into bigger. You’ll then want to regulate these values accordingly.

Establishing the AR Picture Monitoring Configuration

Within the earlier chapters, we used ARWorldTrackingConfiguration to trace the machine’s place and blend the digital content material with the actual world. Apart from the world monitoring configuration, ARKit supplies different kinds of configuration. To carry out picture monitoring, we use a configuration referred to as ARImageTrackingConfiguration. You employ this configuration to detect and observe the movement of identified 2D photos.

To arrange this picture monitoring configuration, replace the viewWillAppear methodology like this:

To precisely detect the place and orientation of a 2D picture in the actual world, ARKit requires preprocessed picture knowledge and information of the picture’s real-world dimensions. The ARReferenceImage class encapsulates this data.

– https://developer.apple.com/documentation/arkit/arreferenceimage

Within the code, we specify to make use of ARImageTrackingConfiguration and inform ARKit what photos to trace. We name the referenceImages methodology of ARReferenceImage to load all referenced photos from the asset catalog. The inGroupNamed parameter takes within the group identify of the AR useful resource group. Recalled that we named it AR Assets, for this reason we move it to the tactic.

As soon as we loaded the reference photos, we assigned them to the trackingImages property of the configuration. Lastly, we name the run methodology to start the picture monitoring operation utilizing the again digital camera.

Overlaying a Video Participant to the Detected Picture

Now that we’ve created the picture monitoring configuration, the query is how can we overlay a video on high of the detected picture?

When ARKit begins the picture monitoring, it routinely provides an ARImageAnchor object when the reference picture is acknowledged. In a Scened-based ARKit app, the renderer(_:didAdd:for:) methodology of ARSCNViewDelegate will probably be referred to as when the anchor is added to the scene. Thus, we will implement the tactic to overlay the video on high of the detected picture.

First, let’s declare the video play for the video playback in ViewController like this:

We load the snowboarding.mp4 file and assign it to AVPlayer for playback. We simply put together the participant for the video playback. The video received’t begin enjoying till we name its play methodology.

Now implement the tactic within the ViewController class like this:

As mentioned, each time a brand new anchor is added to the AR session. The tactic above will probably be referred to as. Subsequently, we first have to test if the newly added anchor is an ARImageAnchor. If it’s a picture anchor, we create a airplane by utilizing SCNPlane for holding the video participant. The scale of the oblong airplane is about to its real-world measurement that you just offered earlier when configuring the reference picture.

Subsequent, we instantiate a SCNNode with the airplane we simply created and add the airplane to the scene by calling addChildNode. Lastly, we name the play() methodology of the participant to start out the video playback.

You could surprise why we have to modify the eulerAngles to rotate the airplane. By default, SCNPlane is vertically oriented. Nonetheless, the detected picture is meant to be horizontal. To match the detected picture’s orientation, we’ve to rotate the airplane. The determine beneath reveals you the default orientation of the airplane.

In case you can’t wait to check out the picture monitoring, you may deploy the app to your iPhone. Assuming you’ve printed out the PDF doc, you may level the machine digital camera on the newspaper. When the app detects the “Snowboarding” picture, it routinely overlay the video participant on high of the picture and begins the video playback.

In case that you just would not have a printer, it’s possible you’ll load the PDF doc on iPad, which has an analogous measurement of the A4 paper. Alternatively, you may open the file in your Mac for testing. However please attempt to resize it to A4 measurement.

Replaying the Video

The picture monitoring ought to work now. Nonetheless, the video playback solely happens as soon as. When the video finishes enjoying, it won’t play once more. To repair this subject, insert the next code within the viewDidLoad methodology:

We added an observer to hearken to the AVPlayerItemDidPlayToEndTime notification, which is posted when the video playback ends. On this case, we set the playback time again to zero.

To play the video once more, insert the next methodology in ViewController:

This methodology is routinely referred to as by ARKit when the anchor is up to date. We first test the reference picture is seen on display after which begin the replay if the result’s true.

Disabling Statistics

You must discover a statistics bar appeared on the backside of the display. If you do not want it, you may disable it by altering the next line of code in viewDidLoad and set its worth to false:

Increase the View to Full Display

By default, the Scene View generated by Xcode has constraints that have been configured respect to the protected space. For this reason you see an empty white bar on the backside of the display. If you wish to make the scene view full display, you turn over to Important.storyboard and alter the scene view’s backside constraint. Within the Attributes inspector, change the Second Merchandise to Superview.Backside and set its fixed worth to 0.

Abstract

On this chapter, we discover one other configuration offered by ARKit that permits you to carry out picture monitoring. With ARImageTrackingConfiguration, it empowers builders to construct some fascinating apps that merges digital content material with the actual world atmosphere. The Harry Potter newspaper is simply one of many many examples. You could apply the method to counterpoint the person expertise and engagement of some real-world objects like postcard and restaurant menus.

It is a pattern chapter of our Intermediate iOS Programming with Swift ebook. To obtain the complete supply code and entry all its content material, you may test it out right here.



[ad_2]

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular

Recent Comments