Software apps and online services
Hand tools and fabrication machines
NOTE: May 2, 2023 - Instructions and code updated to include new AI routines!
BMO-AI is a 3D printed battery-powered companion robot based on a character in the Adventure Time animated series on Cartoon Network that utilizes several different AI capabilities together in order to socialize with people in more engaging and helpful ways.
BMO-AI utilizes a Raspberry PI 3B+ running Raspbian Bullseye and an Adafruit CRICKIT Hat for servomotor control and signal inputs from its button array. BMO has four points of articulation and a 5-inch display for facial expressions and image display using the Pygame library. BMO also uses several other libraries to utilize the following AI capabilities:
- Speech to Text and Text to Speech with offline wake-word detection with Azure Speech Services.
- Single-turn question/answer text completions using OpenAI ChatGPT 3.5 Turbo.
- Muti-turn and contextually aware chat using OpenAI ChatGPT 3.5 Turbo chat completions.
- Image analysis and descriptions using Azure Computer Vision services.
- Image variation “imagination mode” utilizing OpenAI DALL-E 2.
- Raspberry PI Camera Module v3 with image capture and sharing.
- *NEW Paint mode using Stable Diffusion to paint a picture based on your spoken description.
- *NEW DALL-E image from ChatGPT 3.5 description mode where it will create an image based on a description that ChatGPT generates.
***Printedpart files are available at printables.com.***
1. It is recommended that you use PETG or some other filament that is more resilient and temperature resistant than PLA for this build, especially if you do not install a fan to cool the SBC and other components.
2. Aside from the buttons, the model shown in these build instructions used Polymaker Polylite Teal PETG on Prusa MK2 and MK3 printers at.02 layer height.
3. The buttons are labeled by the color they should be printed with, PLA is fine for these.
4. A note about the arms. There are two separate versions of the STLs for the arms, one version (ArmRtFull and ArmLtFull) is modeled for single-part printing, the other versions (ArmRT and ArmLt) are split at 7mm to allow for less support blemishes and will need to be glued together after printing.
5. For the upper and lower legs, you will want to either print without supports, or use support enforcers positioned so that your printer does not print supports inside the tunnel that runs through the leg pieces. Otherwise, you won’t be able to push the string through the leg pieces for the kicking motions.Section 3. Pre-Assembly Instructions
3.1 – Prepping the servomotors
1. After testing your servomotors to ensure they work, you will want to label each one according to their articulation point. For example, use a permanent marker to mark one as LA (left arm) and another as LL (left leg). (Left and right is from BMO’s perspective).
2. After marking your servomotors, you should pre-position the servomotors so that the left servos are at the furthest counterclockwise position (usually 0 deg) and the right servos are at the furthest clockwise position (usually 180 deg). This way all servos start at the lowered position and have range to lift the arms or kick the legs.
3. The servo horns for the arm servos will need to be modified so you can attach the shoulder mounts for the arms. Basically, you will want to use the horn depicted in the image below and cut it down to the point that it will fit inside the printed shoulder mount piece as also shown below.
4. Once the servo horns for the arms are cut so that they slide into the shoulder mounts, you will want to either hot glue or plastic weld those horns into place to secure them to the shoulder mounts, as shown above.
5. Now you can attach the shoulder mounts to the arm servos so that the mount hole for the arm is pointing towards BMO’s back, as shown here. At this point you can screw it into the servo post while gripping the shoulder mount so the servo post doesn’t rotate from that pre-positioned angle.
6. At this point you should remove the servos so that you have room for assembling other parts later.
3.2 - Prepping the button boards
NOTE: You can skip this section if you don’t want functional buttons and plan to just glue the buttons into place instead.
1. First, cut your through-hole protoboards as shown below and use a 5/64 or 3/32 drill bit to create the mounting holes as shown.
2. Loosely mount the boards into position in the main body, then turn the body around and use a permanent marker to put a dot at the center of where each button position is, as shown here. Then remove the boards.
3. Note that there are two prongs on each pole of the switch, so try to align the switches so you can keep track of which prongs you will need to solder for the switch to make the connection when pressed. Then position each switch so that the button itself is centered on each dot you made in the previous step.
4. Now you can solder the buttons and wires onto the board to make a circuit where one pole on each button is connected to a patch wire that will reach the CRICKIT HAT on the Raspberry Pi and the other pole will be connected to a common ground that will plug into the ground pin on the CRICKIT hat signal connectors.
3.3 - Prepping the Speakers
1. Once your button boards are soldered and tested, you’ll want to cut two smaller pieces of through-hole protoboard that are about 5 holes x 5 holes.
2. You’ll then solder the speaker leads and some 2-pin JST or dupont so that the leads are connected to the connector as shown here. These assemblies will be hot glued to the sides of the BMO body so the speakers can be heard through the speaker holes in the body later in the assembly steps.
3.4 - Prepping the Main Body
1. First you’ll probably want to paint the lettering on the sides before you start assembling, it’ll be easier to do it now.
2. Use a soldering iron or an unloaded 3D printing pen to insert the heat-set knurled nuts as shown below:
3. While you’re at it, push some M2 hex nuts into the slots for the servomotor mounts. You really only need to attach the servomotors to the top mount, but you can do the bottom mounts too if you really want to.
3.5 - Prepping the Back Cover
1. Insert three M2 heatset inserts into the back cover as shown below.
2. Insert a square M3 nut into the slot for the battery cover screw hole as shown below, then hot glue it into place after securing it with an M3 screw.
3. Use hot glue or a 3D printing pen to secure the 18650 battery holder, voltmeter, and sliding switch into place as shown here.
4. Mount the 5v voltage regulator to the back cover. Solder the red lead from the battery holder to one switch terminal, then another patch wire that has a female dupont connector which can reach the voltage regulator to the other terminal. Also crimp the ends of the voltmeter and the ground lead from the battery holder with female dupont connectors as well.
5. Connect both the voltmeter and the leads from the battery and switch to the voltage regulator VIN pins. If needed, use hotglue to secure all the leads out of the way on the back cover.
3.6 - Prepping the Raspberry Pi and CRICKIT Hat
NOTE: If you don’t plan on having working buttons hooked up to your board, or plan on using a servo controller other than the CRICKIT HAT, these steps are not necessary.
If you’ve installed the riser terminals for the CRICKIT Hat’s 40 pin connector, remove it. We’ll need the hat to fit on top of the Raspberry Pi with a very low profile if we want the button leads to fit without being squished by the back panel. NOTE: If you plan on putting a heatsink on your Raspberry Pi, use a very low profile heatsink to avoid shorting out the CRICKIT Hat.
2. Attach your long CSI and DSI cables to your Raspberry PI, then slide them through the CRICKIT hat and loosely attach your CRICKIT hat to your Raspberry Pi 40 pin connector. DO NOT push the pins all the way in.
3. Grab one of the printed Raspberry Pi spacers and 14mm M2.5 screw, then slide the spacer into place at the bottom left corner then push your screw in to hold the spacer into place. Use a M2.5 nut to secure that screw.
4. Do the same to the other mounting holes, but do not secure them with a nut, these will screw into the standoffs in the display brackets during assembly. You may also want to crimp a 2-pin female dupont on to one of the right-angle barrel jack tails and install the barrel jack to the CRICKIT power jack at this time as well.
5. IMPORTANT! Now is a good time to hold the assembly and look inside from all sides to ensure that none of the CRICKIT Hat pins on the underside of the board will short out on any part on the top of the Raspberry Pi! If you do notice any pins touching anything, especially a heatsink or the audio jack, disassemble and try using side cutters to trim those contact points.
6. Set this assembly aside for later, making sure that the screws and spacers stay in place.Section 4 - Assembly Steps
4.1 Button and Button Board Assembly
1. Set the main body down face first and place all the buttons in their corresponding slots as shown.
2. Mount the button boards on top of the buttons, as shown.
3. Flip the main body over and check to ensure each button activates its corresponding switch without too much play. If you find that button are activating any switches without being pressed you may need to re-solder the button so that it is flush against the board.
4. Check the length of your button cables, they should reach the center of the monitor hole with some additional play, but not too much.
4.2 – Mount the Display, Speakers, RasPi Assembly, and Camera
1. Install 3 M2.5 hex nuts in the underside of the top and lower right display brackets as shown.
2. Use 5mm M2.5 standoffs to secure the nuts in place.
3. Attach the small fan to the Lower Left Display Bracket with M2 screws in the orientation shown here.
4. Set the display down inside the body over the display hole with the DSI cable connector on the left side. (Yes, this means the display will be upside down.)
5. Secure the monitor with the top and lower right display brackets using M2.5 screws to mount the brackets to the monitor and M3 and M2 screws to mount the brackets to the main body.
6. Without mounting the RasPi assembly, attach the DSI/CSI cables to the monitor and the camera module.
7. Mount the camera module as shown. Note how the top right M2 screw is inserted from the front through the lower right bracket and secured with an M2 nut.
8. Mount the speakers to the sides of the main body so that the speaker covers the speaker holes and the connector boards trail behind them at a bit of an angle.
9. Mount the RasPi assembly to the mountpoints on the Display Brackets, taking care to keep the RasPi spacers in place. (Note: it may be easier to attach any USB connectors before doing this, though I find it’s easier for me to insert them afterwards to keep cables out of my way.)
10. Use hot glue to attach the audio amp above the RasPi assembly on the top wall inside the body as shown here (the wiring will come later).
1. Slide the upper leg parts into the bottom of the main body and try to angle them so that they either bow out slightly or are at least parallel to each other. Then plastic-weld them into place underneath the body. NOTE: It’s best to angle them slightly so that the front of the leg is up against the front edge of the body while they back is further away, causing BMO to lean forward slightly when sitting.
2. Also weld the legs inside the body as well to firmly secure them into place. Use care to make sure you leave the string tunnel opening clear while doing so.
3. Cut two 40cm lengths of string, preferably braided fishing line, and tie one end of each string to the leg servomotor horns with a double or triple knot. (you may need to make one of the horn holes larger to fit the string through, I recommend the second hole closest to the end of the horn).
4. Slide the leg servo motors into place and secure them with M2 screws. Note, you may need to bore the mounting holes of the servos a bit to let the M2 fit.
5. Pass the strings through the tunnel in the upper legs, then through the tunnels of the lower legs until the strings emerge through the bottoms of the feet. Attach the lower legs to the upper legs using M2 screws, making sure not to crimp the lines.
6. With the legs bent to 90 deg and the servomotors positioned so the horn is close to the bottom, make knots in the strings close to the bottoms of the feet, ensuring that the legs remain bent with a slight bit of play in the strings.
7. Plastic weld the holes in the bottoms of the feet so that the holes and strings are not visible.
1. With the shoulder mounts attached, slide the arm servos into position and screw them into place with M2 screws. Make sure the arm mounting hole in the shoulder is pointing towards BMO’s back.
2. Insert the arms into the square shoulder sockets and then secure them with M2 screws. Note: you may need to file or sand the square pegs of the arms down a bit to ensure they slide into the shoulders smoothly, but with little play after screwed in.
1. Ensure that the Servo leads are hooked up to the CRICKIT hat servo pins in the following order (otherwise you’ll have to update the code):
a. Left arm
b. right arm
c. left leg
d. right leg
2. Attach all of the button signal wires to the CRICKIT signal ports with the two common grounds attached to the ground pins.
3. Crimp two pairs of two patch wires with a JST connector for the leads coming from the speaker assemblies and attach them to the audio amp output screw-down terminal.
4. Use patch wires to hook the VIN and ground of the amp to the NEO-PIXEL 5v and ground terminal on the CRICKIT. Also, tie in the power and ground leads from the fan to the same terminal while you’re at it.
5. Create another pair of patch wires, hook those up to the screw-down audio jack and the audio inputs in the Amp, then plug the audio jack into the RasPi audio output.
6. Plug the USB Keyboard jumper and the right-angle USB cable extender into the RasPi USB jacks.
7. Plug the USB Microphone into the USB cable extender, then hot glue that down so that the microphone holes can be seen through the small ports under the directional button on BMO-AI as shown here:
8. Plug the USB power lead into the RasPi USB power jack, then plug the other end into the 5v ouput of the 5v Regulator on the back cover. Also plug the CRICKIT power lead into the other 5v output of the regulator as well. At this point, everything should be plugged in and look somewhat like this:
NOTE: Make sure that the lines into the 5v regulator are oriented so that the grounds are on the inside and the power lines are on the outside pins.
9. Try attaching the back cover to the body while making sure that the larger cables are not in the way of the leg servo horn paths and that no wires are pinched between the body and back cover.
10. Go ahead and secure the back cover to the body with M3 screws.
11. Insert two 18650 batteries into the battery holder then place the battery cover over that and secure that with an M3 screw.Section 5 - Software Setup
1. I highly recommend using the 2022-09-22 release of Raspbian Bullseye as the OS for your SBC in this build. Several required libraries come pre-installed in Bullseye, which makes things a LOT easier here. If you do not use Bullseye, you may need to review the libraries being imported in the code and take steps to ensure all of them are manually installed.
NOTE: I recommend using the user name "bmo" as this will reduce the number of changes you need to make to the code.
2. You'll want to flip the display since the monitor is mounted upside down. This is easy to do by simply open Preferences, select Screen Configuration, then right click the display, select Orientation, then select Inverted. If this does not work, you may need to backrev to the 2022-09-22 Raspbian release as there is a bug with some of the new releases that prevents screen rotation.
3. In the config, you'll need to enable i2c, but will probably also want to enable other things like SSH and VNC.
4. Open the terminal and run the following:
sudo apt-get update
sudo apt-get upgrade
5. In the terminal, set up the CRICKIT hat with the following commands:
pip install Adafruit-blinka
i2cdetect -y 1
6. At this point you should see a matrix with the number 49 inside of it. If you don't make sure your CRICKIT hat is connected properly and not shorting out.
7. In the terminal, run the following to install the Circuit Python library for the CRICKIT:
pip install Adafruit-circuitpython-crickit
5.2 - Set up Azure Speech Services
1. Sign up for a free Azure account at azure.microsoft.com (we’ll be using free tier speech services, so you shouldn’t expect any charges unless you use BMO-AI A LOT!).
2. Create a speech resource in Azure portal using the following steps: https://portal.azure.com/#create/Microsoft.CognitiveServicesSpeechServices
3. Get the keys and region for your resource: https://learn.microsoft.com/azure/cognitive-services/cognitive-services-apis-create-account#get-the-keys-for-your-resource.
4. In the files I share there will be a file that contains the offline keyword model table for the “Hey BMO” wake-word which require no extra steps or code changes to use. However, if you want a different wake-word, you’ll want to create one using the steps at https://learn.microsoft.com/azure/cognitive-services/speech-service/custom-keyword-basics?pivots=programming-language-python.
5. On the terminal, type the following commands:
sudo apt-get install build-essential libssl-dev libasound2 wget
pip3 install azure-cognitiveservices-speech
5.3 Set up Azure Computer Vision
1. In your Azure Portal, create a Computer Vision resource. Make sure to use one of the following regions: East US, France Central, Korea Central, North Europe, Southeast Asia, West Europe, or West US.
2. Use the F0 free pricing tier, it has some usage limitations but should be fine for a single robot like this.
3. Copy the API key and endpoint, you'll need those when you start the coding section.
4. Open a terminal on BMO-AI and run the following commands:
pip install azure-cognitiveservices-vision-computervision
pip install pillow
5.4 Set Up OpenAI
1. Sign up for an account at OpenAI.
2. Copy your API key.
3. Open a terminal on BMO-AI and run the following:
pip install openai
pip install tiktoken
5.5 Install Stable Diffusion libraries
1. Sign up for an account at beta.dreamstudio.ai. and get an API key.
2. Copy the API key.
3. Install the library by typing the following in the console.
pip install stability-sdk
5.6 File downloads and making directories
1. I recommend creating the following directory: /home/bmo/Photos as a place where the photos BMO-AI takes will be stored.
2. Download the main code and the keyword table file to the same directory.
3. Download and extract the face images to the Pictures directory under home.
4. Edit the Python file to add your API keys, settings, and other things as noted in the Code section.Section 6 - Operating BMO-AI
1. Launch the python file by typing python filename.py in a terminal window. If you've done everything right, you shouldn't get any errors.
2. Use any of the following verbal commands after saying the wake-phrase "Hey BMO" (pronounced "be moe").
- Take a photo - Including the word "photo" in your response, (i.e. "Please take a photo of me.") will prompt BMO-AI to take a picture of whatever it is facing and store it in the Photos directory. It will then ask if you want the picture, if you say anything other than no, it will send it to you via email.
- What are you looking at? - BMO-AI will use Azure Cognitive Computer Vision to give you a description of what it sees.
- What are you thinking about? - BMO-AI will take a photo and use DALL-E 2 image variations to show you how it imagines what it's looking at.
- Paint - Including the word "paint" in your response, (i.e. "Please paint a picture of a puppy talking to a penguin.") will prompt Stable Diffusion to create an image based on what you tell BMO-AI to paint.
- Describe - Including the word "describe" in your prompt, (i.e. "Please describe what an orange looks like.") will start the GPT3.5-to-DALL-E description mode where BMO-AI will use ChatGPT 3.5 to describe something, and then send that description to DALL-E to create an image based on the description it provided for some interesting, and sometimes funny, results.
- Let's Chat - Saying "Let's chat" starts the OpenAI ChatGPT 3.5 conversational chat mode where it will remember what your previous questions were in order to have context throughout the conversastion. Saying "I'm done" will end chat mode.
- Ask any question. - If you say anything other than the above key phrases, BMO-AI will use OpenAI ChatGPT 3.5 completions to answer a single question but will not remember what you asked for any follow ups.
3. Note: AI functionality requires an internet connection. If you plan to take your companion robot out on the town outside of your home WiFi range you should set it up to tether to your phone for continued fun on the run. ;)
4. NOTE: Use the voltmeter on the back to determine when you need to shut BMO-AI off and replace the batteries. A general rule of thumb is to shut down between 6.8v and 6.6v in order to prevent reduced battery life due to excessive discharge.