Point and Click for the Real World

Much like everything else in our lives, our homes are becoming more connected all the time. Door locks, speakers, lights, and thermostats — it seems like everything is being connected to the internet to give us new ways to interact with, and automate, these systems. But exactly how we should interact with them is still a matter of debate. There is always the smartphone app, which is reliable, but you have to carry your phone around and deal with unlocking it and opening the app, making it a bit cumbersome. Then there are voice commands, but these systems misunderstand us far too often, and good luck turning on your lights if your internet connection goes down!

As an owner of a number of smart home devices, I think I speak for everyone when I say that we are taking applications for new user interface options! A team at the University of Washington has answered that call with an interesting device called IRIS (Interactive Ring for Interfacing with Smart Home Devices). It is a wireless smart ring that somehow manages to pack in a camera to identify the objects that we are pointing at. When a user of IRIS points, they can then click a button, or perform a simple gesture, to control that object. No apps to unlock, and no misunderstood voice commands in sight.

The system integrates a camera, Bluetooth radio, inertial measurement unit (IMU), and an onboard rechargeable battery. When a user points at a device and presses the ring's button, the onboard camera captures frames of the target device, which are processed on a nearby smartphone. Simultaneously, the user's gesture is captured by the IMU and transmitted for real-time device control.

IRIS is made possible by a number of clever design decisions. First, it addresses the high power demands of camera hardware through hardware and software optimizations, enabling the ring to stream video wirelessly while maintaining battery efficiency for 16-24 hours of operation. Second, it employs an innovative machine learning pipeline for instance-level object detection using contextual scene semantics. By combining YOLO object detection with a centered-object detection algorithm and the DINOV2 model, IRIS achieves precise identification of specific devices within the same class, even, for example, distinguishing between different sets of blinds. This optimization reduces inference latency to as little as 112 ms, ensuring near real-time responsiveness.

The system's performance was validated through extensive real-world testing in various homes and conditions, achieving a high classification accuracy (95 to 98 percent) for device instances and outperforming voice commands in speed and user preference. A user study highlighted its social acceptability and ease of use, positioning IRIS as an innovative alternative to traditional voice commands and smartphone apps for smart home interactions.

There is still some work to be done, however. IRIS does not perform well when attempting to control smaller devices, especially at larger distances, so controlling a lamp across the room, for example, could be a problem. The system is also limited in the set of gestures it can recognize, which would make it a challenge to interact with more complex devices.

But by combining advanced hardware, firmware, and machine learning algorithms, IRIS shows us what is possible in the world of user interfaces. Perhaps one day it will provide us with more natural and efficient smart home interactions.

If smart glasses are more your style, you might want to have a look at ShAIdes.