Smile, You're on Heavy-Handed Camera!

A great deal of effort has gone into developing computational methods that can detect the emotional state of an individual. There is a gap, however, when it comes to understanding emotional states in work environments. Emotional and affective states vary in these environments due to the physical and mental workload, as well as the physiological activity of the worker. Considering that previous research has shown that the emotional state of workers influence factors including job performance, decision making, creativity, turnover, teamwork, and leadership, this is an area that would seem to deserve some attention.

A trio of researchers from the University of Cambridge have recently published their work that seeks to fill in this gap in computational emotion recognition. They have both trained and evaluated machine learning models that are capable of inferring user facial affect while that user is engaged in work-like tasks.

Existing datasets are not adequate for representing the expression of emotional state in work environments, so the team began by collecting a new dataset. Data was collected via multiple cameras, a microphone, and an Empatica wristband that records physiological signals. 12 participants were then observed in environments that simulated an office, a factory production line, and a teleconference call. These participants were asked to perform several tasks that varied in challenge level from easy to difficult and stressful. Finally, a Self-Assessment Manikin questionnaire was given to each participant to self-report their own reactions to each situation and serve as the ground truth labels to pair with the sensor data.

Camera data was processed to extract facial landmarks, which were then fed into a ResNet-18 convolutional neural network. Two additional convolutional layers were added to better represent features for emotional state prediction. The network was pre-trained with 450,000 images from the AffectNet dataset. The team’s own dataset was then used to fine-tune the model’s training.

During testing, the retrained model was found to outperform a ResNet-18 model trained solely on the AffectNet dataset, showing the importance of the purpose-built dataset collected in this study. They also noted that when using spectral representations of the data, performance of the model improved significantly. This shows that capturing facial data over time is critically important for recognizing emotion in work settings.

In the future the team would like to expand the size of their dataset by including groups from multiple European sites. They would also like to explore incorporating additional physiological signals into the analysis. It seems that the researchers have succeeded in their stated goal of recognizing emotional states in the workplace, which may well reap benefits for employers. But the question remains — how will workers respond to having their emotions continually tracked at work, and perhaps being evaluated by it? I suspect that would go over like a lead balloon.