ITYCATT - I'll Teach Your Child A Thousand Things!

Idea 🤔 💡

When I learned about the built-in NFC capabilities of the MATRIX Creator, I thought it could be interesting to create some kind of VUI/NFC hybrid experience. This lead me to the idea of an educational toy for children, which they can interact with both conversationally and physically via NFC-enabled objects. ITYCATT engages children with interactive dialogue, and uses NFC-tagged blocks to teach them shapes, colors, and more! ⭕💬🔵🔺⬛

Getting Started 🔰 👩‍💻

The Matrix Creator is effectively a Raspberry Pi HAT of sorts, mounting to the 40-pin GPIO header found on compatible devices. It requires Raspbian Stretch Lite or Desktop (Buster is not yet supported), instructions for the installation of which are widely available. Once you've inserted your mSD card (flashed using Etcher or similar), it really is as simple as popping the Creator on top, as this handy GIF from the Matrix docs shows:

I chose an Argon ONE case for my particular instance since it includes a temperature-driven fan, which I thought might come in handy once I shoved the whole thing inside a stuffed animal! 😅

In order to get Snips running, I first followed the documented steps to install the MATRIX HAL:

cd ~
curl https://apt.matrix.one/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.matrix.one/raspbian $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/matrixlabs.list
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install matrixio-creator-init libmatrixio-creator-hal libmatrixio-creator-hal-dev
sudo reboot

and Kernel modules:

sudo apt-get install matrixio-kernel-modules
sudo reboot
amixer set PCM 100% && arecord recording.wav -f S16_LE -r 16000 -d 5 && aplay recording.wav

on the Raspberry Pi. Next I installed the Snips SAM CLI tool on my laptop (not the Pi itself) and used it to install Snips onto the remote Pi:

sudo npm install -g snips-sam
sam connect YOUR-RASPBERRY-PI-IP-GOES-HERE
sam init

I then made the documented changes to snips.toml:

sudo nano /etc/snips.toml

mike = "MATRIXIO-SOUND: - (hw:2,0)"

but ran into clipped/non-existent audio output when I tested it:

sam test speaker
sam test microphone
sam install demo

After several weeks of trying suggestions from forum replies etc., MATRIX's own Samreen Islam steered me to a workaround on the MATRIX forums which finally gave me audio output: by removing playback.pcm and adding rate 16000 to pcm.speaker in /etc/asound.conf, resulting in:

pcm.!default {
type asym
capture.pcm "mic"
playback.pcm "speaker"
}
pcm.mic {
type softvol
slave {
pcm "array"
}
control {
name "MicArray Master"
card "MATRIXIOSOUND"
}
}
pcm.speaker {
type plug
slave {
pcm "hw:0,0"
rate 16000
}
}
pcm.array {
type plug
slave {
pcm "hw:MATRIXIOSOUND"
}
}

and then adding portaudio_playback = "default" to [snips-audio-server] in /etc/snips.toml. I repeated the test and demo and confirmed that all audio input and output was at last working as expected:

sam test speaker
sam status
sam test microphone
sam install demo
sam watch

Development Process 💪 💻

I selected the JavaScript MATRIX Lite library in order to get things up and running quickly, as well as integrate easily with Snips' MQTT-based platform. Since the code lives on the Raspberry Pi along with the MATRIX Creator, I also set up Visual Studio Code Remote Development, so that I could write code on my more powerful laptop while executing it remotely on the Pi.

My project takes advantage of the MATRIX Creator's built-in NFC capabilities, so I also had to perform the MATRIX HAL NFC setup, including downloading the NFC Reader Library from NXP's web site, which requires creating an account and logging in before downloading the required zip file. Since this was far more easily performed from my laptop's browser than the Raspberry Pi's, I then scp'd it to the remote Pi:

scp ~/Downloads/SW297940.zip pi@YOUR-RASPBERRY-PI-IP-GOES-HERE:/home/pi/dev/matrix-hal-nfc

From here I cloned and built the wrapper, then tested the example with some NFC-enabled Amiibo to ensure everything was working:

cd ~/dev
git clone https://github.com/matrix-io/matrix-hal-nfc.git
cd matrix-hal-nfc
./install_nxp.sh
./build.sh
cd ~/dev/matrix-hal-nfc/build/examples
./nfc_detect

After extensive setup, and a few complications along the way, it's finally time to write some code! 😅

Actually, there was one last gotcha I wanted to point out; my Raspberry Pi had the latest Node.js, 13.x installed by default - which it turns out has some compatibility issues with the MATRIX Lite NFC JS package - so I used nvm to swap to 12.x, which worked fine:

nvm install 12
nvm use 12

OK, code time, I promise! Now that we've got the right Node.js version, we can install MQTT and the Matrix Lite and NFC packages on the Raspberry Pi:

cd ~/dev
mkdir ITYCATT
cd ~/dev/ITYCATT
npm init
npm install mqtt --save
npm install @matrix-io/matrix-lite --save
npm install @matrix-io/matrix-lite-nfc --save

You can grab the most recent version of the ITYCATT source from GitHub, and if you clone it into ~/dev then running it is as simple as:

cd ~/dev/ITYCATT
node index.js

But wait! There's one other important piece: the Snips Assistant! index.js handles MQTT and the MATRIX Creator functionality, but the intents, slots, wakeword, etc. are managed through Snips. You can create your own Assistant via the Snips Console, but in order to run ITYCATT you need to recreate the required intents and slots, or more easily, just install the Assistant that I've added to the GitHub repo via:

sam install assistant -p assistant

(where the final assistant parameter is the path inside the GitHub repo containing the Assistant files). This is possible since Snips runs purely on your device - unlike Alexa/Google/Cortana/Siri which require the cloud in order to operate. This is important for projects like ITYCATT since no networking is required, meaning you can be confident that your toy isn't spying on you like your phone/computer/smart speaker/TV/thermostat/etc. are! Note that some of the MQTT topics include Snips usersnames, so you'll want to update index.js to use your own if you recreate the Assistant.

Results and Conclusions ⚗️ 📊

The MATRIX Creator's excellent microphone array, built-in NFC reader, and ample on-board sensors make it really easy to explore all kinds of ideas without requiring any external hardware. I was able to develop a powerful, interactive VUI, integrating with Snips over MQTT using Matrix Lite, and even enable interaction with otherwise static objects using the NFC reader. Once I got the basic underpinnings figured out, it was a joy to develop the VUI and other interactive elements in simple JS. I'm really looking forward to incorporating more of the onboard functionality into ITYCATT in the near future! 🚀

Next Steps/Future Enhancements 🔜 🔮

I was only able to realize a fraction of the features that I'd like ITYCATT to have in the provided time. I would like to use the on-board sensors, such as temperature and UV, as well as other environmental factors, such as time of day, to customize the learning experience, e.g. morning vs. bedtime activities, or suggesting that we move away from strong sunlight while playing. I would also like to use the IMU to detect when ITYCATT has fallen over and request assistance. Ultimately I'd like to incorporate an OpenMV Cam H7 in order to provide visual recognition of objects, people, and emotions, enabling for example engaging children as they approach, describing or quizzing them on an item in a particular hand, or asking them to tap a recognized object on the NFC reader! 💡🤯