Reprinted from TidBITS by permission; reuse governed by Creative Commons license BY-NC-ND 3.0. TidBITS has offered years of thoughtful commentary on Apple and Internet topics. For free email subscriptions and access to the entire TidBITS archive, visit http://www.tidbits.com/ ARKit: Augmented Reality for More Than Gaming Josh Centers Apple has a lot of kits. Not model airplanes, but software frameworks with which developers can create apps more easily. You've heard of HomeKit, which developers use to create home automation apps, and we've previously mentioned ResearchKit, which helps programmers write medical research apps. But those two are just the tip of the iceberg ' Apple has ClockKit, CloudKit, GameplayKit, HomeKit, MapKit, PhotoKit, and ReplayKit, to name just a few, and the company announces more at every WWDC. We seldom mention these frameworks because they're usually of interest only to developers. But I think Apple's upcoming ARKit is going to be a big deal for everyone once apps that incorporate it start appearing in iOS 11 (see '[1]iOS 11 Gets Smarter in Small Ways,' 5 June 2017). And I'm willing to bet that we see a flood of them on day one. The AR in ARKit stands for Augmented Reality. In contrast to virtual reality, which aims to encase you in a virtual world, augmented reality blends the real and virtual worlds by overlaying digitally generated images on live video. The most famous example of AR is the smash-hit Pokémon Go app, which I wrote about a year ago in '[2]What the Heck Is Pokémon Go?' (17 July 2016). How ARKit Works -- Augmented reality isn't new, and it wasn't new when Pokémon Go hit the scene. Examples of AR have been around since the dawn of smartphones, but it has long been a relatively crude technology. For instance, the [3]Amazon app can show you what a TV will look like on your living room wall, but you have to tape a dollar bill to the wall so the app knows where to position the TV. The [4]InkHunter app can give you a preview of how a tattoo will appear on your arm, but it needs you to draw a smiley face where you want it to go. Those apps need the dollar bill and the smiley face as placeholders because they can't recognize the three-dimensional objects in your photos. That's what Apple hopes ARKit can fix. ARKit analyzes camera and motion data to recognize surfaces and construct planes with which digital objects can interact. Additionally, ARKit can leverage all that data to apply the correct lighting to digital objects. In short, ARKit makes it so digital objects can interact with real-world surfaces and their appearance will change based on actual lighting conditions so they blend in as naturally as possible. In this [5]video, a robot dances around a living room, landing perfectly on the floor with each step. Note how the reflection from the lamp light follows the robot around. IFRAME: [6]http://www.youtube.com/embed/JQvwrGePd7Q Is your mind blown yet? Hold on, there's more. A developer used ARKit to create a portal between virtual and real worlds. In the [7]video, the developer walks through the virtual door, around a virtual world, and looks back out at the real world. IFRAME: [8]http://www.youtube.com/embed/rIPfpGCxONQ One last example. Check out this [9]video. It's just two guys playing basketball, right? Look closer. The players have been inserted into the scene. IFRAME: [10]http://www.youtube.com/embed/lQt96saECfM Remember that we're in the earliest stages of ARKit testing and development. Developers are just getting started. Maybe you're impressed, but you may also be thinking, 'That's neat, but what use is it?' While many of the initial applications may be for gaming, there will be plenty of useful applications. ARKit in the Real World -- After Apple releases iOS 11, tape measure apps will be the new flashlight apps ' expect a bunch of them. There are already several working concepts, but this [11]video shows the most impressive I've seen yet. IFRAME: [12]http://www.youtube.com/embed/z7DYC_zbZCM But this concept can be taken further. Here's a [13]video of an app that can measure the square footage of an entire room. IFRAME: [14]http://www.youtube.com/embed/iw9MPZoPqCQ Speaking of rooms, as someone who has spent all summer moving furniture around, I'm excited by this [15]video of a concept app by game developer Asher Volmer, which lets you use AR to place furniture in a room. IFRAME: [16]http://www.youtube.com/embed/DSYeN624ick Playing with ARKit -- If you're an Apple developer, you can experiment with Apple's [17]ARKit demo app, which lets you place a few simple objects in AR. You'll have to use Xcode to compile the project and install the app, but it's not too onerous if you know what you're doing. I'm no programmer and I was able to get it working. If you don't want to fuss with the ARKit demo app, I'll give you a taste. It works a lot like the Camera app. The app scans for surfaces onto which it can place objects, and when it recognizes one, it puts a yellow square on it. The center button lets you choose an object to place: a candle, cup, vase, lamp, or chair. Once you've placed an object, it acts like a 3D object in real space. [18][tn_ARKit-place-candle.jpg] The chair makes for an even better demonstration. I can walk around it just as though it were a real chair. [19][tn_ARKit-place-chair.jpg] I can even put my foot under the chair. [20][tn_ARKit-foot-under-chair.jpg] ARKit apps will have a lot of potential uses around the house. Instead of moving all your furniture around to figure out where it fits best, you could use ARKit to place objects around the room, saving yourself a lot of work and back pain. Likewise, ARKit will appear in all sorts of shopping apps. Imagine being able to use AR to place a sofa in your living room before ordering it. One app [21]shows what this could be like, using a pillow as an example. IFRAME: [22]http://www.youtube.com/embed/bwNAykf4IKw Alper Guler shows how ARKit could assist in helping us decide what dish to order at a restaurant. Ever glanced over at other diners to see what their dishes looked like before ordering? What if you could see every dish from the menu right in front of you at the table? Here's a [23]video showing that. IFRAME: [24]http://www.youtube.com/embed/oFdgVNg4ryM ARKit will also prove useful when traveling in an unfamiliar city. This demo [25]video shows how an app can pair ARKit with CoreLocation to give you AR views of waypoints, as well as AR direction lines you can follow as you walk down the street. IFRAME: [26]http://www.youtube.com/embed/6Lo0Z7CkMWw The more I use ARKit, the more I believe that Apple has to be thinking about taking it into products that go beyond the iPhone and iPad. Walking down the street holding an iPhone in front of your face to see how your real world has been augmented will be awkward (as we know from Pokemon Go players). It's begging to be integrated into a pair of glasses. We've had a taste of that, with Google Glass. While the initial technology was promising, it suffered from bugs and somewhat overblown privacy concerns (it's not like people weren't constantly taking pictures and recording video with their phones). However, Glass's second life in the industrial world (see '[27]Google Glass Returns'¦ In Factories and Warehouses,' 19 July 2017) may indicate that the initial release of Glass simply lacked the killer app. With the kinds of augmentation possible with ARKit backing it up, Apple may be able to create electronic eyewear that would be both functionally compelling and socially acceptable. References 1. http://tidbits.com/article/17274 2. http://tidbits.com/article/16622 3. https://itunes.apple.com/us/app/amazon-shopping-made-easy/id297606951?mt=8&at=10l5PW 4. https://itunes.apple.com/us/app/inkhunter-try-tattoo-designs-in-augmented-reality/id991558368?mt=8&at=10l5PW 5. http://www.youtube.com/watch?v=JQvwrGePd7Q 6. http://www.youtube.com/embed/JQvwrGePd7Q 7. http://www.youtube.com/watch?v=rIPfpGCxONQ 8. http://www.youtube.com/embed/rIPfpGCxONQ 9. http://www.youtube.com/watch?v=lQt96saECfM 10. http://www.youtube.com/embed/lQt96saECfM 11. http://www.youtube.com/watch?v=z7DYC_zbZCM 12. http://www.youtube.com/embed/z7DYC_zbZCM 13. http://www.youtube.com/watch?v=iw9MPZoPqCQ 14. http://www.youtube.com/embed/iw9MPZoPqCQ 15. http://www.youtube.com/watch?v=DSYeN624ick 16. http://www.youtube.com/embed/DSYeN624ick 17. https://developer.apple.com/arkit/ 18. http://tidbits.com/resources/2017-07/ARKit-place-candle.jpg 19. http://tidbits.com/resources/2017-07/ARKit-place-chair.jpg 20. http://tidbits.com/resources/2017-07/ARKit-foot-under-chair.jpg 21. http://www.youtube.com/watch?v=bwNAykf4IKw 22. http://www.youtube.com/embed/bwNAykf4IKw 23. http://www.youtube.com/watch?v=oFdgVNg4ryM 24. http://www.youtube.com/embed/oFdgVNg4ryM 25. http://www.youtube.com/watch?v=6Lo0Z7CkMWw 26. http://www.youtube.com/embed/6Lo0Z7CkMWw 27. http://tidbits.com/article/17348 .