AR and VR Researcher Jason Orlosky Gets Gaze-Tracking Working in Python — for Under $20

Augmented and virtual reality (AR and VR) researcher Jason Orlosky has released a tool that lets you track exactly where an eye is looking — using a single Python script and an affordable off-the-shelf cameras.

"It's a heavily optimized port of the C++ repository released in 2017 by me, Yuta [Itoh], Leszek [Swirski], and Alex. Unlike the pupil fitter I released last year, this tracker provides the 3D origin and gaze direction for use with many eye tracking applications. Note that this is a first release, and I will continue to optimize as time goes on."

Eye-tracking, which allows a system to determine where the user is gazing, can be used with great effect in virtual and augmented reality: it can allow for rendering effort to be focused at the center of the user's vision while putting less work into the periphery, it can be used to adjust field-of-view on the fly, and it can even be used as a control system — choosing menu options by looking at them and blinking, for example.

Orlosky and colleagues had already developed a working 3D eye-tracking system, written in C++ and released under a permissive license — but this new version makes two key changes. The first is that the $100 infrared eye-tracking camera previously recommended has been replaced with a much more affordable $17 model, attached to a pair of cheap sunglasses via a pipe cleaner. The second is that everything you need is contained in a single Python script.

"Gaze origin and direction are printed directly on the OpenCV output window," Orlosky explains, "which you can also write directly to file from the Python script if desired."

The source code for the project is available on GitHub under the permissive MIT license; it should run on any system with a free USB port, Python 3, the opencv-python, tkinter, and numpy libraries, plus PyOpenGL if 3D visualization is required.