AR You Ready to Rock?

Kevin Lin loves to play the piano, but he hates to carry around stacks of sheet music. The engineer in him couldn’t help but come up with a technological solution to this problem. Most people would probably choose to load their sheet music onto a lightweight tablet and call it a day, but not Lin. Oh no, that would somehow still be too cumbersome. Lin’s idea was to use augmented reality glasses to display the musical notes. That way, he would never even have to look at the sheet music, it would be right in front of his eyes.

As you can imagine, someone that does not want to go to all the trouble of looking at sheet music on a tablet might be inclined to procrastinate when it comes time to build a better mousetrap. So Lin sat on the idea for a few years, but finally, the stars aligned. A local hackathon hosted by Mentra, a maker of smart glasses, not only gave him the motivation to bring the idea to life, but also the chance to win a free pair of smart glasses.

Once Lin got going, the build actually proved to be pretty simple. Sheet music was downloaded in the MusicXML format, which is a rich language for describing music. Next, a Python library developed at MIT, called Music21, was leveraged to render the MusicXML files into images. A little bit of optimization in ImageMagick was the last step necessary before displaying the images in a pair of Mentra Mach 1 smart glasses.

To select a particular piece of music, voice commands can be issued to scroll through a system of menus. The glasses show just a portion of the music at a time. The next bit was initially loaded by pressing a foot pedal, but Lin found that to be difficult to manage and instead modified the system to load the next segment after a predefined period of time, which can be sped up or slowed down using the pedals.

While the device does technically work, Lin said that it does not work well for his application. The main problem is that there is a three-second delay between initiating the transfer of an image and it showing up on the display. That can be a very long time when playing the piano. Maybe future optimizations could fix that, or maybe not. But at least Lin got the idea out of his head — oh yeah, and a cool pair of smart glasses.