Mojing Mojing - A Smart Mirror with ReSpeaker!

Something I’ve always wanted is a magic mirror. An advisor who’s near omniscient that I can ask for information. Growing up I imagined it being similar to the magic mirror from Shrek, witty, having great showmanship, and the ability to be a friend.

Well, I’m technically no longer a child. I’m 28 (10 at heart), and I’ll settle for what I can access around me and what my team can create. Technology has made this relatively simple to throw together. This project isn’t what’s describe above – it lacks the wit, could use some better showmanship, and is not omniscient (yet!) – but, it’s definitely real and not just part of our stories and imaginations. And the best part? It’s largely open source. That’s right, you can grab this yourself and create it – I hope with some changes and improvements you are willing to share.

Task 1: Configuring the Hardware

Step 1) Preparing the Raspberry Pi 3

We started with software – we wanted to know if we could get an existing project to work with Chinese without using a VPN, as this project will first be shown at the Taobao Maker Festival in Hangzhou, China, and thus all online services would have to be allowed through the Great Firewall. We do have plans to add English and other languages in the future as well.

We found and open source project: Magic Mirror. It fit our needs quite well as it can run on the Raspberry Pi 3 Model B+ (you can also use the Raspberry Pi 3 Model B), which is small and we can fit it behind the mirror to hide it.

The best part is how easy it is to add in 3rd party modules, which they already have a lot of.

1a) Download stuff

Download this customized Raspberry Pi image which has several additional packages pre-installed to run the ReSpeaker mic arrays. Burn the image to your SD card. We recommend rufus or ether.Note: Do not use the lite version – the lite version does not contain a GUI, which we obviously want with this project.

1b) Figure out how to connect to your Raspberry Pi.

If you have an extra keyboard, great! Plug it in, and you should have a screen you will be using with the mirror! If you do not have an extra keyboard, or you do not yet have the extra screen, you can setup the WiFi configuration and enable SSH before the first time boot. Add a file named ssh to the boot partition of the SD card – this will enable SSH. Next create a file named wpa_supplicant.conf with the following content – however, make sure you replace the ssid (network name) and psk (passcode) with your own!

country=GB
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
     ssid="WiFi SSID"
     psk="password"
}

1c) Attach the ReSpeaker Linear Array Pi HAT to the Pi.

Simple, but go ahead and attach the ReSpeaker Pi HAT to the Raspberry Pi now. You will need it to test the voice assistant later. Plus I should plug our product at this point, right?

1d) Connect to the Raspberry Pi.

Power on your Raspberry Pi and connect. If you are using SSH then use the Pi’s IP or raspberry.local (requires mDNS support – Windows users will need to install Bonjour). Then you can login via SSH. We recommend the good ol’ PuTTY for Windows users.

Step 2) Installing and Configuring Magic Mirror

Now we need to install the magic mirror software and get that set up on our Raspberry Pi. For this step you will need a monitor – preferably the one you intend to use. We are going to use two 3rd party add ons: MMM-Remote-Control, so we can remotely control the mirror, and MMM-kalliope an assistant framework we will use. We chose Kalliope as some other personal assistant options already integrated with the Magic Mirror project had some issues operating in China.

2a) First let’s clone & install the project into ~/MagicMirror with the following command.

bash -c "$(curl -sL https://raw.githubusercontent.com/MichMich/MagicMirror/master/installers/raspberry.sh)"

This also will install other dependencies that we need, such as node and npm.

Note: Do not use apt install to install node and npm, node and npm in the deb repository is kind of outdated. Remove them if already installed.

2b) Install the Magic Mirror Modules MMM-Remote-Control andMMM-kalliope.

cd ~/MagicMirror/modules
git clone https://github.com/kalliope-project/MMM-kalliope.git
git clone https://github.com/Jopyth/MMM-Remote-Control.git
cd MMM-Remote-Control
npm install

2c) Update Magic Mirror’s configuration file, located at ~/MagicMirror/config/config.js to include these modules. I like nano, but feel free to use whatever editor you want.

{
    module: "MMM-kalliope",
    position: "upper_third",
    config: {
        title: "",
        max: 1
    }
},
{
    module: 'MMM-Remote-Control'
    // uncomment the following line to show the URL of the remote control on the mirror
    // , position: 'bottom_left'
    // you can hide this module afterwards from the remote control itself
},

2d) Next restart Magic Mirror (we just powered on/off the Raspberry Pi) and test to see if kalliope is working properly with the following command:

curl -H "Content-Type: application/json" -X POST -d '{"notification":"KALLIOPE", "payload": "my message"}' http://localhost:8080/kalliope

2e) Magic Mirror has a default weather module installed. It is based on OpenWeatherMap. You will have to register and place your API key in ~/MagicMirror/config/config.js.

Step 3) Installing a Voice Assistant

Here is where you may split off. Our demo is made for a Chinese audience, so we use a Chinese speech service. If you can read Chinese, head over here. If not, I will do my best to translate to English.

The voice SDK is called 问问 (wenwen) which translates to “ask”. This can support offline speech recognition, as well as online.

3a) Clone voice-engine/wenwen.

cd ~git clone https://github.com/voice-engine/wenwen.git
cd wenwen

3b) Next, we will download the wenwen Linux SDK, then extract lib and .mobvoi into the wenwen directory.

 wenwen
 ├── .mobvoi
 ├── lib
 ├── assistant.py
 ├── offline.py
 └── player.py 

3c) We can run offline.py to recognize commands such as “turn on the light”, “turn off the light”, "play music", or other such commands.

3d) To use the online mode you will want to register at https://ai.chumenwenwen.com/ and paste your key into wenwen/assistant.py.Then run assistant.py and say “你好问问” (ni hao wenwen), which translate to “Hello Ask”, to wake up the assistant and ask questions.

Step 4) Add in Questions and Responses / Open Door with Wio Link

I wanted to do more than just add voice. I wanted Mojing to actually interact with the world and not just restrict it to the same area, but anywhere. So I chose to use our Wio Link which simply allows us to use an API call to interact with the world as long as both Mojing and the Wio Link have WiFi. If you are not familiar with Wio Link check out its wiki. I will assume that you have already set up your Wio Link and have the access token and API call. The wiki for Wio Link has some tutorials that explain this.

4a) Let's add in a voice command first. We'll use 打开门, which means open the door. To do this we will have to make sure we are in the ~/respeaker directory, and run these commands:

[ ! -d ~/respeaker ] && git clone https://github.com/respeaker/respeaker_for_raspberrypi.git ~/respeaker
cd respeaker
git pull origin master
cd magic_mirror_zh
ln -s ~/wenwen . 

4b) Next we need to edit mirror.py. Here is a direct link to the file on github - I'd rather not paste it all on this project page.

We need to edit class Mirror and create a new key word, "打开门".

def __init__(self, key=KEY):
       super(Mirror, self).__init__(key)
       self.set_keywords(['开灯', '关灯', '播放音乐', '几点了', '暂停', '魔镜', '显示天气', '隐藏天气', '打开门'])
       # self.set_keywords(['魔镜'])
       self.kws = False
       self.message_bus = MessageBus() 

We need to add ourself into the set_keywords, which as you can see above we did. These keywords translates to:['lights on', 'lights off', 'play music', 'What is the time', 'pause', 'magic mirror', 'show weather', 'hide weather', 'open the door'].

That's great! We now have a keyword to open the door.

4c) Next we need to have they keyword perform an action. So we need to edit part of this Mirror class again, this time the on_partial_transcription definition.

def on_partial_transcription(self, text):
       if self.kws:
           if text.find('魔镜') >= 0:
               # self.message_bus.put('>'')
               self.kws = False
               self.recognizer_start()
           elif text.find('打开门') >= 0: 
               # open door
               self.message_bus.put('打开门')
               open_door()
           elif text.find('开灯') >= 0:
               # turn on light
               pass
           elif text.find('关灯') >= 0:
               # turn off light
               pass
           elif text.find('显示天气') >= 0:
               # show weather info
               self.message_bus.put('{"action": "SHOW","module":"module_4_currentweather"}', 'REMOTE_ACTION')
           elif text.find('隐藏天气') >= 0:
               # hide weather info
               self.message_bus.put('{"action": "HIDE","module":"module_4_currentweather"}', 'REMOTE_ACTION')
       else:
           self.message_bus.put(text)
       print('on_partial_transcription: {}'.format(text))

This will display the text 打开门 (open the door) on the screen - that's what the self.message_bus.put function does - and then call a function called open_door(). I guess we should make that function...

4d) The function is actually simple. You will want to put your Wio Link's access token in the code as well.

# Use Wio Link to open a door
def open_door():
   # TODO: fill your Wio Link's access token
   access_token = ''
   cmd_template = 'curl -k -X POST https://us.wio.seeed.io/v1/node/GenericDOutD1/onoff/{}?access_token={}'
   cmd = cmd_template.format(0, access_token) + " && sleep 1 && " + cmd_template.format(1, access_token)
   subprocess.Popen(cmd, shell=True)

In this code we call the Wio Link API and tell it to unlock the door, wait one second, the lock it.

cmd = cmd_template.format(0, access_token) + " && sleep 1 && " + cmd_template.format(1, access_token)

cmd_template stores the API call.

Congratulations!That's it. We are done with the code and configuration. All we have to do now is create the frame! This is...

Task 2) Building a Frame

Now we have to build the frame, and mount the electronics. This isn’t too difficult if you have the right tools.Alternatively, we have seen people modify existing mirror frames to work just fine. We are not experts in this area, so good luck! The official Magic Mirror website has some build instructions as well.

Step 1) Preparing a Screen

1a) Choose your screen. We chose an older monitor we had laying around the office. It is a bit heavy, but it worked out fine. If you can fine newer screens/monitors that weigh less, and have the money to burn, I’d advise that.

You may have noticed I said screen and monitor. That’s because you will be taking apart the monitor to just get the screen and electronics out. You can actually find some screens online that do not have the bezel and other casings. These tend to be half the price or less than a normal monitor and usually have some small, and sometimes unnoticeable, defect. That’s a great economical route, and could save you some work. Here are, in my opinion, the top three criteria to consider when you are choosing your screen:

i) Size – This determines the smallest size your mirror can be. Note that you can have the screen only take up part of the mirror.

ii) Brightness – This allows you to view the screen’s content when there is ambient light in the environment. The mirror will dim the output of the screen, so this is quite important if your environment is well lit.

iii) Resolution – If you want to play YouTube videos or display a lot of information you will want a higher resolution screen.

1b) Remove the screen from the monitor. Be careful not to damage the screen. This will also expose the high voltage electronics, so be cautious.

Step 2) Measure and Order the Mirror and Frame Components

2a) Measure the screen and the electronics. Don’t forget to measure length, width, and height. As I did my research a few people had issues with the height/depth of the electronics and fitting them into mirrors. Also Be sure you know how you will mount these and handle cables. They quickly get in the way.

2b) Order your mirror. We ordered two in case we broke the original, or the chance to build a second smart mirror. You can choose between glass or acrylic. You can read a lot more about this on the magic mirror forums.

2c) Order wood. Always order a bit extra – if you’re like me you will make several mistakes. I always forget to measure twice and cut once. Be sure you compensate for the thickness of the screen, electronics, and the mirror. Also don’t forget to leave some room or path for heat dissipation - these can get quite warm.

Step 3) Build the Frame

Create your frame. I won’t go into too much detail here. We are not the best with wood working skills – you can find sources for this elsewhere – I’ll show a few photos however. I do wish I had the initial drawing I gave to my coworker on the whiteboard. However, we didn’t save this. Also, be sure to use safety gear, and never forget, “measure twice, cut once.” Just some advice that may seem obvious think about how you want to route your cables, were you want your mic array to be located, and how heavy this will be once finished. Honestly, I never considered the weight of an older screen, so it’s quite hefty and will be difficult to take to multiple events.

Note: When the machines were on, we did have protective gear.

Step 4) Mount the Electronics

Next, we had to mount the electronics. We just threw then onto a an acrylic piece we had lying around.

Step 5) Mojing Assemble!

Assemble and test your mirror. Now that we have a frame we can assemble it all together and power on. We had a few issues with a loose capacitor on our older screen. After we had moved it around we could hear an audible crackling sound as it would charge and discharge, causing our screen to blackout now and then. Fortunately, I was able to grab one of our test engineers to troubleshoot and fix this issue. Aside from this issue everything booted up correctly.

Step 6) Building the Wio Link Demo

We created a box, and used a rubber band to tension it to open if it was not locked. To lock it we used the Grove Electromagnet.