Translation Toy

Updated for 2019 China-US Young Maker Competition

Learning a new language is difficult, but there are those that believe children around the ages of 6 or 7 are at the peak age for learning a second language (although I haven't personally read any of the studies that suggest this. Sounds good though ¯\_(ツ)_/¯). Given this, I decided to throw together a toy that could translate anything said to it in one language and repeat it back in a second language since it seemed like a fun learning project.

Hardware

I ended up starting off with a small animatronic Porg toy from Star Wars that I picked up on sale at Toys R Us back when they were closing.

It consists of three major parts that we can take advantage of:

• a speaker
• a front button
• a set of motors controlled by a single ground/voltage wire combination for moving the mouth and wings

There's also a portion with the sound controller, on/off switch and batteries screwed on to the bottom, but this project won't require that part.

After removing the bottom cover, we can see that the wires from the base are connected to the main parts that we will want to use. I ended up just cutting these wires and soldering on extensions.

At this point I wasn't 100% sure if the motor would work by just applying a voltage, or if the board under it was doing something more complicated with PWM, so I hooked the motor wires to 5V and ground off of an Arduino. Luckily, it worked.

Now that I knew everything worked as I was expecting, it was time to hook in the first stage of actual hardware. I decided to use a Raspberry Pi with Google's AIY Voice Kit hat since it gave me an easy to use microphone and speaker driver using a built in MAX98357A. It's worth noting that I'm using a V1 Voice HAT, rather than the bonnet, since I wanted to be able to connect it to a monitor, keyboard, and mouse without having to find my USB and HDMI adapters, so the pins used will reflect that.

For this setup I hooked the two speaker wires from the Porg into the speaker connectors on the hat, the button wires into the GPIO24 and ground connectors under the 'Servos' heading, and for the motors I needed to hook up a relay for a 5v connection driven by the signal off of GPIO26. The microphone that came with the AIY Voice Kit was attached to the board using the pre-defined mic connector.

Setting Up Google Translate + Code

Rather than going into all of the details on how to create a GCP project and create your service key, I'm going to refer to the Before You Begin section of the official documentation, as this process changes in small ways fairly regularly.

After you create a service account and get your JSON key file, install the google-cloud-translate library for Python on your Raspberry Pi.

pip install --upgrade google-cloud-translate

At this point it's time to set up the Python script that will handle controlling the toy and translating what it hears. Before we begin, I will say that I'm not the most familiar with Python and working with Raspberry Pis, so this was definitely a chance for me to try something simple and work on that.

The top of the file will include all of our imports that will be used in this script

import aiy.audio
import aiy.cloudspeech
import aiy.voicehat
import os
from gpiozero import LED
import RPi.GPIO as GPIO
from google.cloud import translate 

You can see that we're relying on the AIY library for all audio aspects of this project. I happened to have that library pre-installed on my SD card image since I used Google's AIY image. You can find all of the related info for that image or your own setup on GitHub here. I also ended up using the LED package from gpiozero for driving the motors, as I just needed a simple on/off for triggering the relay, and I liked the syntax.

Under the imports you will need to set your GOOGLE_APPLICATION_CREDENTIALS value so that the script is able to communicate back with Google's translation services. This is the same JSON file that you downloaded for your service account.

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="/home/pi/Downloads/TranslationToy.json"

Next, initialize the translation client and the toy's motors under GPIO26 (again, I did this as an LED to control the relay)

translate_client = translate.Client()
porg = LED(26)

Now it's time to write the meat of the script. You will start by creating a main() method and setting all of the initial values necessary for listening for a button press and recognizing speech.

def main():
   GPIO.setmode(GPIO.BCM)
   GPIO.setup(24, GPIO.IN, pull_up_down=GPIO.PUD_UP)
   recognizer = aiy.cloudspeech.get_recognizer()
   aiy.audio.get_recorder().start()
   print('ready')

We will then use a loop to check for button state changes, and then act based on that. Every time the script enters the loop, I ensure that the motors are not running. This can probably be optimized, but it works for now :P

   while True:
       porg.off()

We can check if the button was pressed by seeing if a signal has gone to ground. If it has, we can start listening to the user.

       if not GPIO.input(24):
           print('Listening...')
           aiy.i18n.set_language_code('en')
           text = recognizer.recognize()

After we've verified that the user has said something, we can turn the audio recording into text and send it off for translation. Once we receive that translation, we can repeat it back to the user. For this example, we're expecting anything said to the toy to be said in english (en), though we could expand on the project and allow this to change later.

           if not text:
               print('Sorry, I did not hear you.')
           else:
               porg.on()
               print('You said "', text, '"')
               target = 'es'
               translation = translate_client.translate(
                   text,
                   target_language=target)
               aiy.i18n.set_language_code('es-ES')
               aiy.audio.say(u'{}'.format(translation['translatedText']))

You'll notice that even though we set the target for the translation to Spanish (es), we're still setting the AIY language code to Spanish, as well. This is because the device doesn't handle accents and pronunciation correctly if we leave it in English mode.

Finally, we'll need to make sure that our main() method is called when the script starts.

if __name__ == '__main__':
   main()

At this point we can press the button in the toy and say something simple, like "Hello, how are you?" and it will repeat it back to us in Spanish.

Or we can try any level of complex conversation to have it translated, thanks to the Google translation service backing the project.

Cellular Connection

I had some people interested in this project for teaching children, but an issue that was brought up is that a wireless connection isn't always going to be available. To get around this, I decided to play with using a cellular connection as well. Using a Soracom SIM and Huawei MS3121 USB modem, I finally found a way to get a network connection working over cellular via this tutorial. While it's a lot slower of an option, it does handle the issue of having a connection available when it's needed.

Next Steps to Make It Better

Since this was more of a "let's see if I can make this work" project, there's definitely a lot of room for improvement and new features. Next I'd want to add functionality for detecting the spoken language, rather than relying on just English, and I'd want to add a command to change the output language. Currently the translation/speech engine used supports dozens of languages, so it'd be easy enough to switch the output to something else, such as Mandarin from Cantonese, or French to English in parts of the US or Canada. I'd probably also create an app that allows you to change the device languages without saying anything, that way the user can avoid getting stuck in a language mode that they don't know how to get out of. I also thought about adding a camera so users could say "What is this?" and it'd take a picture, classify it using machine learning, and then say what the object is in a different language. There's definitely a lot of potential for some fun new features on this toy, but overall it's a good practice project for something that could be useful.