Are You Doing Anything Later?

Keeping tabs on what activities a person is engaged in throughout the day has many practical applications, ranging from fitness and rehabilitation tracking to real-time task assistance and education. Considering how useful this information can be, many attempts have been made to improve the state of the art in automated human activity recognition. Of these efforts, sound has proven to be one of the most useful signals — whether one is washing their hands, brushing their teeth, or otherwise, these activities generate very distinctive sound patterns. But using audio signals comes with a few problems that has thus far limited where and when technologies leveraging them can practically be put to use.

Since we generally want to recognize any activities that we are engaged in throughout the course of the day, wherever they may take place, that means a practical device needs to be portable, and preferably wearable. But devices in this form factor have limited compute power and battery life, which is not appropriate for the relatively high processing requirements of an always-on audio processing tool. Moreover, privacy concerns abound when capturing a constant stream of audio from an individual, and users are unlikely to accept this from an activity tracker. A new approach taken by a team of researchers at Carnegie Mellon University may alleviate these concerns, however. They have developed a method that uses low-resolution audio data in conjunction with an inertial measurement unit (IMU) to substantially reduce computational resource requirements and protect privacy, all while maintaining a high level of accuracy in recognizing human activities.

The team's method, called SAMoSA (Sensing Activities with Motion and Subsampled Audio), relies upon a power-optimized IMU sampling motion measurements at 50 Hz. When the tracker detects a possible activity of interest using only the IMU data, a microphone is then turned on and its data is fed into an algorithm that considers both audio and motion data to classify human activity. To reduce the computational workload, audio is subsampled directly at the hardware level, at a rate of no more than 1 KHz. This also has the effect of preserving privacy — at this level of resolution, human speech is garbled and cannot be understood. Since the IMU uses very little energy, this sensor combination allows for always-on functionality without rapidly draining a small, wearable battery.

An implementation of SAMoSA was created using an Apple Watch. Measurements from the IMU were fed into a lightweight Random Forest algorithm to detect the start of an activity of interest, which would then trigger the microphone to start collecting audio samples. From there, both audio and IMU data were passed into a multimodal deep learning classification model that was trained to recognize a battery of 26 common human activities. A study was conducted to assess the performance of the model and it was found that it was capable of classifying activities correctly over 92% of the time across several different indoor environments.

To prove the concept, the system used an external laptop to run the machine learning algorithms. However, the team believes that their models could run on a smartwatch by making a few optimizations, like compressing the model and using floating point quantization. At this time, the researchers are also focused on expanding SAMoSA to recognize more than the initial 26 activities. There may be challenges as they go down that path, as many activities will prove to be similar to one another and difficult to distinguish between. But if these areas can be adequately addressed, SAMoSA may prove to be a great option for practical, always-on human activity tracking in the near future.