The Human Side of Computing

Facial expressions convey a wealth of nonverbal information in our everyday interactions with others. When it comes to interacting with machines, on the other hand, this information is often lost, as it can be simpler and less expensive to deal with other inputs, such as voice. This makes human-machine interactions less rich, and also closes the door to many biometric and security-related applications that are also possible with the use of facial expression information.

One major barrier preventing more widespread adoption of facial expression data into devices is the lack of an adequate imaging system. The best candidates are 3D light-field cameras, which are very small due to their microlens arrays situated between an objective and an image sensor, and are also capable of rendering very high-precision depth reconstructions. However, due to optical crosstalk between neighboring microlenses, these cameras suffer from image contrast problems that obscure the fine details needed to recognize facial expressions.

A team of researchers at the Korea Advanced Institute of Science and Technology have recently designed a new light-field camera that side steps image contrast issues by operating in the near-infrared range. Whereas existing cameras of this sort require additional bulky hardware consisting of external sensors and lighting units, this new device has packaged everything — including the vertical-cavity surface-emitting lasers for illumination — into a single, tiny package that could easily be incorporated into a smartphone.

The team’s camera was evaluated to determine its suitability for facial expression classification applications. Facial light-field data was captured from a cohort of 32 adult subjects who were instructed to demonstrate expressions of happiness, anger, sadness, and disgust. Next, the OpenFace software package was utilized to identify the locations of thirteen different facial landmarks. Even using the new device, the cheek and forehead areas were still saturated due to high levels of specular reflection, and so were excluded from the landmark analysis.

A multilayer perceptron classifier was trained on this dataset, with the goal of recognizing facial expressions in light-field data. On average, the classification accuracy was observed to reach a very respectable 85%. This result is a substantial improvement over approaches that only use 2D facial information, as would be the case with a traditional camera.

Without some further work, you will not see this technique being used outside of a research lab any time soon. At present, the depth reconstruction algorithm takes 10 minutes to process on a computer with an Intel i5 3.60 GHz CPU and 16 GB of RAM. The researchers theorize that this algorithm may be able to run dramatically faster if redesigned for processing on a GPU, with the massive parallelization that would then be possible. In any case, the novel light-field camera developed by this team looks poised to improve human-computer interactions in the future.