CheekyKeys Lets You Code by Moving Your Face
Unlike traditional dictation software, CheekyKeys allows users to input keystrokes by mouthing Morse code.
The idea
The YouTuber and maker known as Everything Is Hacked recently became a dad, and with that, came several challenges when he had to do work on his laptop. Often, he would have limited use of his hands, owing to holding a baby, and using the more traditional verbal dictation software would be far too noisy. So instead, he wanted to find a way to use his face to input text data into his computer without the need for speaking, and naturally, Morse code was an ideal choice.
A first attempt
To begin this endeavor, Everything Is Hacked began exploring various strategies for capturing facial expressions. Initially, he tested a simple Python script that used OpenCV in combination with Dlib to track eye movements and print the letter 'x' whenever a blink was detected. However, blinking multiple times in such quick succession can be quite disorienting and even lead to motion sickness, so an alternative was needed. Ultimately, he decided that opening and closing his mouth was far easier, and it freed him up to implement shortcuts as well by making small movements with his eyebrows.
MediaPipe
The libraries Everything Is Hacked had been using weren't ideal for people with beards and/or glasses, so he instead turned to Google's machine learning project MediaPipe, which offers a very detailed model that maps to hundreds of facial points. A quick change to the Python script meant that the model now classifies a video frame, outputs whatever gesture is present, and causes the corresponding keyboard key to be pressed virtually.
Putting it to the test
With everything now ready, the system was put through its most challenging task yet: allow a coding interview to be passed without pressing a single key on the keyboard or moving the mouse. As seen in Everything Is Hacked's video on YouTube, the process ended up being quite hilarious, owing to the multitude of exaggerated facial expressions. But in the end, he did well, and plans on practicing to get even faster in the near-future.