Are You Looking at Me?

EyeMU lets you interact with out-of-reach widgets on your smartphone screen with just a glance and a gesture.

Nick Bild
21 days agoMachine Learning & AI
Using EyeMU (📷: A. Kong et al.)

As smartphones have grown larger — sometimes even blurring the line between phone and tablet—the extra real estate on the screen has been put to good use. But that larger screen can make one-handed operation more challenging, with on-screen items being further apart than the thumb can reach. This puts users in the awkward position of having to shift the position of the phone in their hand in order to reach certain areas of the screen, or use a second hand. It is safe to say that many phones have met sad ends by being dropped while shuffling them around with a single hand.

With this problem in mind, researchers at Carnegie Mellon University have developed a method that allows users to access any portion of a smartphone’s screen without having to physically touch it. Called EyeMU, this method uses a combination of user gaze tracking and simple hand gestures to interact with those widgets on the screen that always seem to be just out of reach.

The initial prototype of EyeMU was developed as a javascript application that runs in real-time in the Safari browser on Apple iOS devices. However, they envision that future versions would run natively at the OS-level, where they would provide an API such that any application could implement the functionality.

To prevent accidental misfiring of EyeMU triggered actions, a multistep process has been implemented. The first step in the pipeline is locating the user’s face using the phone’s front-facing camera. Once located, the software then verifies that the user’s gaze is focused on the screen. MediaPipe's Face Mesh, which is a machine learning-based tool that detects 3D face landmarks in real-time on mobile devices, is used in both of these initial steps.

After the initial criteria has been satisfied, the next step is to determine exactly where on the screen the user is looking. A convolutional neural network (CNN), trained on the GazeCapture dataset, was used to estimate gaze location. After converting the CNN to a TensorFlow JS model, the team was able to process twenty frames per second on an iPhone 12 Pro. If a user’s gaze is found to be within a 2.5 centimeter diameter circular region for a duration of 500 milliseconds, it is considered to be fixed in that region.

Once gaze fixation has been detected, EyeMU then waits for a gesture to be performed. To detect gestures, data from the inertial measurement unit is fed into a Support Vector Classifier. For the early trials, a set of seven gestures (plus a neutral pose) were defined. These gestures include simple motions such as flicking left or right, or pulling the phone closer to the face. The screen widget that the user is gazing at is acted upon in the way that has been defined for the detected gesture. This could be, for example, dismissing a notification by looking at it then giving a leftward flick of the hand.

Before EyeMU is ready for commercial use, the accuracy of the gaze module will need to be improved — while the mean error of 1.74 centimeters is fairly impressive, it is still not sufficient for smartphone use where sub-centimeter accuracy is often important. Further, EyeMU has not been tested on a wide range of devices; in theory, it should work on many phones, but that has yet to be verified. It is also worth noting that the processing required by the method draws a substantial amount of power from the battery, which could limit the time that such features could be used. Limitations aside, EyeMU shows a lot of promise as a new way of interacting with large-screen phones without adding any new hardware. Perhaps future advancements to both EyeMU and smartphone technology will make the technique more practical for everyday use.

Nick Bild
R&D, creativity, and building the next big thing you never knew you wanted are my specialties.
Latest articles
Sponsored articles
Related articles
Latest articles
Read more
Related articles