Smile! You're on Doppler Camera!

The myriad benefits of smart homes appeal to many, but to add the smarts, environmental sensing is a prerequisite. This sensing typically is achieved with the help of cameras or microphones. But in many, if not most, cases, this is considered by users to be too intrusive. To expand the use cases for smart home applications, an alternate sensing technology is needed.

One very promising candidate to fill this role is millimeter wave Doppler radar. It can offer much the same richness of data as can be acquired with cameras or microphones, but does not compromise privacy in the same way. Sounds great, right? As you might expect, it is not all smooth sailing — whereas there is a huge corpus of existing audio and video data sets available to train machine learning models, the same cannot be said for radar data.

Rather than undertake the huge, expensive effort of creating a massive library of sensor data, a team at Carnegie Mellon University had the idea to leverage existing video data by converting it into realistic, synthetic Doppler radar data of human activity.

The team’s software pipeline first computes the 3D positions of key points on the human body in each frame of video using the VIBE adversarial learning framework for human pose estimation. Next, nine different viewpoints are simulated from the 3D key points to make the final machine learning model more view-invariant. The remaining steps of the pipeline filter and smooth this data as they synthesize the final whole-body Doppler signal.

The method was tested out on a VGG-16-based convolutional neural network that was trained on over ten hours of video data that had been converted to synthetic Doppler radar signals. This classifier was trained to recognize twelve different human activities. Real-time data was captured with a Texas Instruments AWR1642 RF Doppler sensor and fed into the trained model. Of the 720 activities performed by ten participants, the model achieved an accuracy of 81.4%. This is not quite as good as the better than 90% achieved when training a model on native sensor data, but shows that this new method is very effective at leveraging existing video data nonetheless.

Doppler radar data hits the sweet spot between data richness and privacy protection in many respects. Considering that Doppler sensors are also inexpensive, ranging from a few dollars at the low end, to a few tens of dollars for more sophisticated devices, it is easy to see them being integrated into a whole host of smart home and medical sensing applications in the near future.

There are still some hurdles to clear before this technique can be used by a wider audience, however. Presently, it is required that input videos have a static background — the model would fail if the background or camera were in motion. Further, while there are no images taken of users of the system, the users’ activities are still logged, which in and of itself has implications for privacy. Even with these limitations in mind, this new method represents a step forward for many activity recognition applications.