How To Build A Super Quick Smile Tracking App

Facial tracking has become an increasingly popular feature in mobile apps, powering everything from playful selfie filters to sophisticated avatar animations. At the heart of many of these experiences is the ability to detect and respond to the user‘s facial expressions – especially smiles!

As a full-stack developer specializing in augmented reality, I‘ve seen firsthand how even a simple smile tracking app can delight users and drive engagement. There‘s just something magical about seeing an app react to your emotions in real-time.

In this in-depth guide, we‘ll walk through how to build a basic but powerful smile tracking app using ARKit, Apple‘s framework for augmented reality. Whether you‘re a beginner looking to explore AR or an experienced developer wanting to add face tracking to your app, this tutorial will provide a solid foundation.

The Power of a Smile

Before we dive into the technical details, let‘s take a step back and consider why smile detection is so compelling. Smiling is one of the most universally recognizable human expressions, transcending language and culture. It‘s a signal of happiness, friendliness, and approachability.

In the context of an app, detecting smiles opens up a world of possibilities for more emotional and engaging user experiences. Some potential applications include:

• Social media filters that adapt to the user‘s expression
• Games that use smiles as a control input
• Mental health apps that track mood over time
• Dating apps that recommend matches based on smile patterns
• Camera apps that automatically capture smiling selfies

The list goes on and on! As AI continues to advance, we‘ll likely see even more innovative uses of facial expression analysis.

Of course, it‘s important to consider the privacy implications of any app that collects and analyzes biometric data like faces. Developers should be transparent about what data they collect, how it‘s used, and provide easy ways for users to opt-out.

How Face Tracking Works

So how does ARKit actually detect facial expressions like smiles? At a high level, it uses a combination of computer vision and machine learning algorithms to analyze the raw depth data from the front-facing TrueDepth camera found on newer iPhones and iPads.

When a face is detected, ARKit fits a generic 3D mesh model to match the contours of the user‘s face. This mesh consists of a set of vertices in 3D space that approximate the shape of facial features like the nose, eyes, mouth, and cheeks.

Here‘s an example of what the raw mesh data looks like, courtesy of Apple:

As the user‘s expression changes, ARKit updates the position of the mesh vertices in real-time. By analyzing the relative positions of certain vertices, we can infer what expression is being made.

ARKit provides a higher-level interface to this mesh data in the form of blend shapes. Blend shapes are pre-defined deformations of the mesh that correspond to common facial expressions like smiles, frowns, eyebrow raises, and so on.

For each blend shape, ARKit provides a coefficient value between 0 and 1 indicating the relative strength of that expression. A value of 0 means the expression is not present at all, while 1 represents the maximum intensity. These coefficients are what we‘ll use to detect smiles in our app.

Building the App

Now that we have a basic understanding of facial tracking, let‘s get into the nitty gritty of building the app! Feel free to follow along with the provided code samples or use them as a reference for your own implementation.

Prerequisites

To build this app, you‘ll need:

• A Mac running macOS 11 or later
• Xcode 12 or later
• An iOS device with a TrueDepth front camera (iPhone X or later, iPad Pro 11-inch or later) running iOS 14 or later
• Basic familiarity with Swift and ARKit

Step 1: Setting Up the Xcode Project

First, open Xcode and create a new project. Choose the "iOS" tab and select the "App" template. Give your project a name like "SmileTracker", choose "Storyboard" for the interface and "Swift" for the language, then save it to your desired location.

Next, we need to configure the project to use ARKit. Go to the project settings, select the app target, and click the "+" button in the "Signing & Capabilities" tab. Search for "Face ID and Attention" and add that capability.

This will automatically add the necessary face tracking entitlement to your app.

Finally, open the Info.plist file and add the "Privacy – Camera Usage Description" key. Provide a short description of why your app needs camera access, like "This app uses the camera to track your facial expressions."

Step 2: Checking for Face Tracking Support

With the project set up, let‘s start writing some code! Open the ViewController.swift file and add the following imports at the top:

import UIKit
import ARKit

This gives us access to the ARKit framework. Next, add the following property to the ViewController class:

let sceneView = ARSCNView()

ARSCNView is a special view that renders the live camera feed with AR content overlaid on top. It‘s the core component we‘ll use to display the face tracking data.

In the viewDidLoad() method, add the following code:

guard ARFaceTrackingConfiguration.isSupported else {
  fatalError("Face tracking is not supported on this device")
}

This checks if the current device actually supports face tracking. If not, we abort the app with a fatal error. You may want to handle this more gracefully in a production app, like showing an alert and providing alternative features.

Step 3: Requesting Camera Permissions

In order to use the camera for face tracking, we need to request the user‘s permission. Add the following code after the face tracking support check:

AVCaptureDevice.requestAccess(for: .video) { granted in
  guard granted else {
    fatalError("Camera access was not granted")
  }

  DispatchQueue.main.async {
    self.setupFaceTracking()
  }
}

This uses the AVFoundation framework to request camera access. If granted, we dispatch to the main queue to set up the face tracking session (we‘ll implement that next). If denied, you should provide a friendly message prompting the user to enable camera access in settings.

Step 4: Setting Up Face Tracking

Now let‘s set up the actual AR face tracking session! Add the following method to the ViewController class:

func setupFaceTracking() {
  let configuration = ARFaceTrackingConfiguration()
  sceneView.session.run(configuration)
  sceneView.delegate = self

  sceneView.frame = view.bounds
  view.insertSubview(sceneView, at: 0)
}

Here‘s what‘s happening:

1. We create an instance of ARFaceTrackingConfiguration, which tells ARKit to track the user‘s face in real-time.

2. We run the face tracking session on our ARSCNView instance.

3. We set the view controller as the delegate of the ARSCNView so we can receive updates.

4. We set the ARSCNView to fill the entire screen and insert it as a subview behind any other UI.

To act as the AR delegate, we need to conform to the ARSCNViewDelegate protocol. Add the following extension to the end of the ViewController.swift file:

extension ViewController: ARSCNViewDelegate {
  func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
    guard let faceAnchor = anchor as? ARFaceAnchor else { return }

    let leftSmileCoefficient = faceAnchor.blendShapes[.mouthSmileLeft] as! Float    
    let rightSmileCoefficient = faceAnchor.blendShapes[.mouthSmileRight] as! Float

    let averageSmileCoefficient = (leftSmileCoefficient + rightSmileCoefficient) / 2
    DispatchQueue.main.async {
      self.handleExpression(smileCoefficient: averageSmileCoefficient)
    }
  }
}

This is where the magic happens! The renderer(_:didUpdate:for:) method gets called up to 60 times per second with data about the detected face.

We first cast the ARAnchor argument to an ARFaceAnchor, which gives us access to the blend shape coefficients. We extract the values for the left and right sides of the mouth smiling, and calculate the average between them.

Then we dispatch to the main queue to update the UI based on the smile intensity (we‘ll fill in that logic shortly).

Step 5: Responding to Smile Intensity

Let‘s implement the handleExpression(smilingCoefficient:) method that gets called with each face tracking update. Add the following code to the ViewController class:

func handleExpression(smileCoefficient: Float) {
  let emoji: String

  switch smileCoefficient {
  case _ where smileCoefficient > 0.8:
    emoji = "😄"
  case _ where smileCoefficient > 0.6: 
    emoji = "😊"
  case _ where smileCoefficient > 0.4:
    emoji = "🙂"
  default:
    emoji = "😐" 
  }

  showEmojiLabel(with: emoji)
}

func showEmojiLabel(with emoji: String) {
  let emojiLabel = UILabel()
  emojiLabel.text = emoji
  emojiLabel.font = UIFont.systemFont(ofSize: 120)
  emojiLabel.textAlignment = .center
  emojiLabel.frame = CGRect(x: 0, y: 0, width: 240, height: 180)
  emojiLabel.center = view.center
  view.addSubview(emojiLabel)
}

The handleExpression(smileCoefficient:) method uses a switch statement to determine which emoji to display based on the intensity of the user‘s smile. The showEmojiLabel(with:) method creates a simple label displaying the emoji in the center of the screen.

Feel free to customize the emojis and intensity thresholds to your liking! You could also display custom images or trigger animations instead of emojis.

Step 6: Cleaning Up

There‘s one final step before we can test the app – we need to clean up the AR session when the view controller is dismissed. Add the following code to the viewWillDisappear(_:) method:

override func viewWillDisappear(_ animated: Bool) {
  super.viewWillDisappear(animated)
  sceneView.session.pause()
}

This pauses the AR session when the view is no longer visible, freeing up device resources.

And that‘s it! Build and run the app on your ARKit-compatible device. Grant camera access when prompted, then point the front camera at your face. You should see the emoji change in real-time as you change your expression!

Taking It Further

Congratulations, you‘ve built a fully-functional smile tracking app in just a few dozen lines of code! But this is just the tip of the iceberg when it comes to the power of ARKit‘s face tracking.

Some ideas to extend the app further:

• Customize the UI with branded assets or animations
• Use the smile intensity to control other aspects of the app, like sound effects or haptic feedback
• Capture a photo when the user‘s smile reaches a certain threshold
• Track other facial expressions like frowns, blinks, or winks
• Integrate with a backend API to collect smile frequency analysis over time

With a little creativity, you can create truly engaging and emotionally intelligent apps. Just be mindful of the performance implications of any additional features, as the face tracking algorithms are already quite computationally intensive.

If you‘re interested in diving deeper into the technical details of ARKit face tracking, I highly recommend watching the "Understanding ARKit Face Tracking" session from WWDC 2018. It provides a great overview of how the blend shape coefficients are calculated and how to use the raw face mesh data.

I also recommend checking out Apple‘s Creating Face-Based AR Experiences sample code project for more advanced techniques like overlaying 3D content on the user‘s face.

The Future of Face Tracking

As powerful as today‘s facial tracking technology is, it‘s still just the beginning. Rumor has it that future Apple AR glasses will feature even more advanced eye tracking and expression analysis. Imagine being able to navigate an app hands-free, using only subtle eye movements and facial gestures!

Even without dedicated AR hardware, facial tracking has the potential to revolutionize how we interact with our devices. As front-facing depth cameras become more common, we‘ll likely see more apps leveraging facial data to create hyper-personalized experiences.

Of course, with great power comes great responsibility. Developers and tech companies must prioritize the ethical collection and use of biometric data, with clear consent and privacy safeguards in place. Apps that analyze faces should aim to enhance the user experience, not exploit personal information for profit.

One area where I see enormous positive potential for facial tracking is mental health. There‘s already some fascinating research being done on using machine learning algorithms to detect signs of depression, anxiety, and other conditions based on facial cues. While this technology is still in its infancy and should not replace professional diagnosis, it could one day provide a valuable early warning system and help connect people to the resources they need.

Personally, I‘m excited to continue experimenting with the creative and technical possibilities of facial tracking. It‘s a field that combines cutting-edge computer science with deep knowledge of human emotion and psychology. The breakthroughs we make here could have a profound impact on how we interact not just with machines, but with each other.

Conclusion

In this tutorial, we‘ve covered the basics of using ARKit to track facial expressions in real-time. We walked through the steps to set up a new Xcode project with face tracking capabilities, request camera permissions, detect smiles using blend shape coefficients, and respond to that data in a simple user interface.

Along the way, we‘ve touched on some important concepts like the TrueDepth camera system, the AR face mesh, and performance considerations for AR apps. We‘ve also explored some of the current and future applications of face tracking technology, from selfie filters to mental health monitoring.

I hope this guide has given you a solid foundation to start building your own face-based AR apps. The field of facial tracking is still relatively young and there‘s so much exciting research and development happening every day. As a full-stack developer specializing in AR, I truly believe this technology has the potential to make our digital experiences more humane and empowering.

If you have any questions or feedback on this tutorial, feel free to reach out to me on Twitter @tundsdev. You can also find the complete source code for the SmileTracker app on GitHub. Happy coding!