Published December 18, 2022 © MIT

BD-1: vision AI powered droid

Star Wars BD-1 droid powered by Raspberry Pi and vision AI, assembled from LEGO bricks.

IntermediateFull instructions providedOver 3 days10,365

Things used in this project

Hardware components

Raspberry Pi 4 Model B

LEGO Star Wars 75335 BD-1

Max7219 8x32 dot matrix LED display

M5Stack Servo Kit 180° Brick-compatible

Webcam Aukey PC-W1 1080p 30fps

Software apps and online services

Raspberry Pi Raspbian

Microsoft Azure

Story

Introduction

BD-1 is an exploration droid first appearing in "Star Wars Jedi: Fallen Order". Recently we saw BD droid in "The Book of Boba Fett" series, and it was no surprise LEGO decided to release a set featuring the droid.

I am a Star Wars fan, and one of my favorite droids is BD-1, so I decided to build it and empower it with AI. In the following short video, you can see the final result.

Video demo showcasing BD-1 droid

In a nutshell, this project will show you how to:

build a realistic BD-1 droid from two LEGO sets and some additional parts
use Raspberry Pi to control the LED panel
utilise Azure Custom Vision with Raspberry Pi and camera
control droid head based on object detection

LEGO assembly

LEGO set 75335 was released earlier this year, and the BD-1 droid you can assemble from it is approximately 30cm in height. If you want to fit Raspberry Pi and maybe even a power bank into the droids head you should build it larger, and you will need two sets. Also, with two sets you will have the opportunity to make the droid higher and more realistic.

LEGO Star Wars 75335 BD-1

Assembly will require a few additional parts, I used some parts from the LEGO Mindstorm set. Building instructions you get in the set can be used as a guide, but feel free to make modifications as you believe it is best, after all, it is your droid.

With an LED display on the back of the head and all the extra parts you add there will be extra weight, so keep in mind legs need to hold that weight, so make them strong and stable.

BD-1 body assembly

USB connector and power adapter take some extra space when they are plugged into the Raspberry Pi, I first tried positioning it on the side, but in the end, I had to put it horizontally in the head.

BD-1 head assembly

The camera can be easily placed as part of one eye, and cable goes behind it into the head and Raspberry Pi. This will also give you enough space to place a small power bank into the head.

If you feel like you need some extra LEGO bricks you can check the "Pick a Brick" shop and order specific bricks. Usually, it takes a few weeks for bricks to arrive so plan in the advance. I have also found a few 3D modeling tools for LEGO enthusiasts, but I had a feeling that creating a whole model with such tools would take a lot more time than actually assembling it.

Raspberry Pi

Raspberry Pi is pretty easy to set up, if you need any help there is a great tutorial on the official site. Once it is up and running you should open the configuration.

sudo raspi-config

Make sure to enable SPI (Serial Peripheral Interface) because you will be using pins to control the LED display and motor.

Raspberry Pi configuration

I used a standard USB camera, when connected you can check it is detected with vcgencmd command.

vcgencmd get_camera

This command should give you the result "supported=1 detected=1"which means the camera is detected and ready for use.

LED display

On the back of the head, I used a MAX 7219 LED display that consists of 4 modules with 64 separate LEDs. All LEDs are addressable and you can easily create patterns you like with them.

MAX 7219 LED display

The display is connected to the Raspberry Pi in the following way:

MAX7219 LED display pin schema

Testing display with Python can be done with the following code, you can also set intensity via the code. The full example is attached and available on GitHub.

# create matrix device
serial = spi(port=0, device=0, gpio=noop())
device = max7219(serial, cascaded=n, block_orientation=block_orientation, rotate=rotate, blocks_arranged_in_reverse_order=inreverse)

# set initial intensity
intensity = 16
device.contrast(intensity)

Addressing a single LED is done with the draw point command, where x and y are the exact addresses of the LED you want to light on.

draw.point((x,y), fill="white")

To make back pannel lights realistic I created a pattern with a matrix, moving odd rows in one direction and even in the other.

Servo control

M5Stack has a nice servo kit with 2 motors, which I found perfect for my need. The motor is positioned on the droid's neck, turning the head left and right.

M5Stack 180 Servo Kit

The motor is connected to Raspberry Pi pins 4, 8, and 9.

M5Stack 180° servo pin schema

With the following lines of code, you can test the motor and put it in starting position. Don't forget to run stop and cleanup in the end.

import RPi.GPIO as GPIO
from time import sleep

GPIO.setmode(GPIO.BOARD)

# servo motor is connected on pin 8
GPIO.setup(8, GPIO.OUT)

servo=GPIO.PWM(8, 50)
servo.start(0)
sleep(1)

# stop and cleanup
servo.stop()
GPIO.cleanup()

With ChangeDutyCycle command motor can be placed in different positions in the following way:

# left (-90 deg position)
servo.ChangeDutyCycle(5)
sleep(1)

# neutral position
servo.ChangeDutyCycle(7.5)
sleep(1)

# right (+90 deg position)
servo.ChangeDutyCycle(10)
sleep(1)

If the motor is shaking after the positioning, you can run ChangeDutyCycle with a value of zero to make it stop.

servo.ChangeDutyCycle(0)

Depending on the motor's position this can be fine-tuned later if needed, once you have vision AI running and want to make the turning angle correct.

Azure Custom Vision

With the use of the camera I wanted to make droid detect objects, so I decided to use Azure Custom Vision. Create a new project, define the Resource Group, set the project type to Object Detection, and the domain to General.

Azure Custom Vision new project

Add images to your new project, you can take them with the camera connected to Raspberry Pi, or use your mobile phone if you find that is easier for you.

Azure Custom Vision adding images

Before training the model all images should be tagged correctly, clicking on the image and drawing an area on it will set the tag.

Azure Custom Vision tagging images

The process of taking images and tagging them can be time-consuming. In my free time, I work on a web-based tool that allows synthetic image generation for vision AI model training, and the tool integrates with Azure Custom Vision. With the use of a realistic 3D model, I was able to generate a few hundred images and automatically tag them in a matter of seconds. The tool is called syntheticAIdata, it is open for signups if you are interested to check it out once it gets released.

With images ready for training, you can press the "Train" button in the top right corner of the Azure Custom Vision interface, and in a few minutes, your model will be trained.

Azure Custom Vision training model

The last step needed is to publish the model in Azure Custom Vision, so you can access it via API. URL and keys are found in the project settings.

Vision AI

The vision AI model we trained is able to detect R2D2, so we want to test it out. The whole code is on GitHub, here are a few highlights.

First, we need to capture the image from the camera and we will use OpenCV. If it is not installed yet, you can install it with pip in the following way:

pip install opencv-python

OpenCV needs to be imported into your Python code and we can capture the image.

camera = cv2.VideoCapture(0)
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 320)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 240)

ret, image = camera.read()
cv2.imwrite('capture.png', image)

The image from the camera is stored in the capture.png file. The image is validated with our model and we get an array of results. In case we have one R2D2 in the image this array will have one element.

with open("capture.png", mode="rb") as captured_image:
results = predictor.detect_image("<PROJECT_ID>", "<ITERATION_NAME>", captured_image)

for prediction in results.predictions:

Calculating the R2D2 location in the image is done in the following way:

locationBD1 = prediction.bounding_box.left + prediction.bounding_box.width / 2

Based on the location we can turn on the motor and move the head in the desired direction.

The result of the detection can be drawn on the picture and stored with the following code:

bbox = prediction.bounding_box
result_image = cv2.rectangle(image, (int(bbox.left * 640), int(bbox.top * 480)), (int((bbox.left + bbox.width) * 640), int((bbox.top + bbox.height) * 480)), (0, 255, 0), 3)
cv2.imwrite('result.png', result_image)

If there was an object detected in the image, your result will look like this:

Vision AI model R2D2 object detection

As you will notice in the code I position BD-1 head based on 3 main positions depending on where R2D2 is. It is also possible to position it in the exact location based on the calculation listed above.

Final words

I had a lot of fun building this project, and I hope I also inspired you to build something cool. Hopefully, this project gave you some useful tips on it.

And remember: recycle your droids, save the galaxy!

Code

# BD1 LEGO Max7219 controller
# Author: Goran Vuksic

import time
import argparse
import random
import numpy as np

from luma.led_matrix.device import max7219
from luma.core.interface.serial import spi, noop
from luma.core.render import canvas

def bd1(n, block_orientation, rotate, inreverse):

    # create matrix device
    serial = spi(port=0, device=0, gpio=noop())
    device = max7219(serial,
                     cascaded=n,
                     block_orientation=block_orientation,
                     rotate=rotate,
                     blocks_arranged_in_reverse_order=inreverse)

    # set initial intensity
    intensity = 16
    device.contrast(intensity)

    # initial pattern
    matrixbd1 = [
    [[1,0,1,0,1,1,0,1],[1,1,0,1,1,1,0,1],[1,1,1,0,0,1,0,1],[0,1,1,0,1,0,1,1]],
    [[0,1,1,0,1,0,1,1],[0,1,1,1,0,1,1,0],[0,1,1,1,1,1,0,1],[1,1,0,1,1,0,1,1]],
    [[1,1,0,1,1,1,0,1],[1,1,0,1,1,1,0,1],[1,1,0,1,1,0,1,1],[1,1,0,1,0,1,1,0]],
    [[1,1,1,0,1,1,1,0],[1,1,1,0,1,0,1,1],[1,0,1,1,0,1,1,1],[1,1,1,0,1,1,0,1]],
    [[1,0,0,1,1,1,0,1],[1,1,0,1,1,0,1,1],[0,1,1,1,1,0,1,1],[1,1,0,1,1,0,1,1]],
    [[0,1,1,0,1,0,1,1],[1,1,1,1,1,0,1,1],[1,1,1,0,1,1,1,0],[1,0,1,1,1,1,1,1]],
    [[1,1,0,1,1,1,0,1],[1,1,1,0,1,1,0,1],[1,0,1,1,0,1,0,1],[1,1,0,1,1,0,1,0]],
    [[1,1,0,1,0,1,1,1],[1,0,1,1,1,1,0,1],[1,1,0,0,1,1,0,1],[0,1,1,0,0,1,1,1]]
    ]

    # main loop
    while True:
        # turn on lights
        with canvas(device) as draw:
            for i in range (4):
                for j in range (8):
                    for k in range (8):
                        if matrixbd1[j][i][k] == 1:
                            draw.point((k,j + 24 - 8*i), fill="white")

        # sleep
        time.sleep(0.2)
        
        # change intensity
        intensity = random.randint(1, 8) * 2
        device.contrast(intensity)
        
        # shift matrix
        for a in range (8):
            for b in range (4):
                if a % 2 == 0:
                    matrixbd1[a][b] = np.roll(matrixbd1[a][b], -1)
                else:
                    matrixbd1[a][b] = np.roll(matrixbd1[a][b], 1)
        for a in range (8):
            if a % 2 == 0:
                templastled = matrixbd1[a][0][7]
            else:
                templastled = matrixbd1[a][3][0]
            if a % 2 == 0:
                for b in range (3):
                    matrixbd1[a][b][7] = matrixbd1[a][b + 1][7]
            else:
                for b in range (3, 0, -1):
                    matrixbd1[a][b][0] = matrixbd1[a][b - 1][0]
            if a % 2 == 0:
                matrixbd1[a][b + 1][7] = templastled
            else:
                matrixbd1[a][0][0] = templastled
        
               
        
if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='bd1_max7219 arguments',
        formatter_class=argparse.ArgumentDefaultsHelpFormatter)

    parser.add_argument('--cascaded', '-n', type=int, default=4, help='Number of cascaded MAX7219 LED matrices')
    parser.add_argument('--block-orientation', type=int, default=0, choices=[0, 90, -90], help='Corrects block orientation when wired vertically')
    parser.add_argument('--rotate', type=int, default=1, choices=[0, 1, 2, 3], help='Rotate display 0=0°, 1=90°, 2=180°, 3=270°')
    parser.add_argument('--reverse-order', type=bool, default=False, help='Set to true if blocks are in reverse order')

    args = parser.parse_args()

    try:
        bd1(args.cascaded, args.block_orientation, args.rotate, args.reverse_order)
    except KeyboardInterrupt:
        pass

# BD1 LEGO Controller
# Author: Goran Vuksic

import cv2
import RPi.GPIO as GPIO
from time import sleep
from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
from msrest.authentication import ApiKeyCredentials

# credentials for Azure Custom Vision
credentials = ApiKeyCredentials(in_headers={"Prediction-key": "<PREDICTION_KEY>"})
predictor = CustomVisionPredictionClient("<ENDPOINT_URL>", credentials)

GPIO.setmode(GPIO.BOARD)

# servo motor is connected on pin 8
GPIO.setup(8, GPIO.OUT)

# start servo
servo=GPIO.PWM(8, 50)
servo.start(0)
sleep(1)
# position it in neutral position
servo.ChangeDutyCycle(7.5)
sleep(0.5)
servo.ChangeDutyCycle(0)
sleep(0.5)

# camera
camera = cv2.VideoCapture(0)
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 320)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 240)

# positions: 0 - left, 1 - middle, 2 - right
currentposition = 1

try:
    while True:
        # store image
        ret, image = camera.read()
        cv2.imwrite('capture.png', image)

        with open("capture.png", mode="rb") as captured_image:
            results = predictor.detect_image("<PROJECT_ID>", "<ITERATION_NAME>", captured_image)

        for prediction in results.predictions:
            if prediction.probability > 0.5:
                print (prediction.bounding_box)
                locationBD1 = prediction.bounding_box.left + prediction.bounding_box.width / 2
                if locationBD1 < 0.3:
                    if currentposition != 0:
                        servo.ChangeDutyCycle(5)
                        currentposition = 0
                elif locationBD1 > 0.7:
                    if currentposition != 2:
                        servo.ChangeDutyCycle(10)
                        currentposition = 2
                elif locationBD1 > 0.3 and locationBD1 < 0.7:
                    if currentposition != 1:
                        servo.ChangeDutyCycle(7.5)
                        currentposition = 1
                sleep(0.5)
            servo.ChangeDutyCycle(0)
        # enable to store result as image with bounding box 
        #bbox = prediction.bounding_box
        #result_image = cv2.rectangle(image, (int(bbox.left * 640), int(bbox.top * 480)), (int((bbox.left + bbox.width) * 640), int((bbox.top + bbox.height) * 480)), (0, 255, 0), 3)
        #cv2.imwrite('result.png', result_image)

except KeyboardInterrupt:
    GPIO.cleanup()
    camera.release()

Credits

Goran Vuksic

4 projects • 25 followers

Engineering manager, Microsoft AI MVP, cofounder of syntheticAIdata, father, hitchhiker through the galaxy...

BD-1: vision AI powered droid

Things used in this project

Hardware components

Software apps and online services

Story

Introduction

LEGO assembly

Raspberry Pi

LED display

Servo control

Azure Custom Vision

Vision AI

Final words

Schematics

How to connect Max7219 and M5Stack servo

Code

Max7219 controller

Servo controller

Vision controller

Github

Credits

Goran Vuksic

Comments

Embed the widget on your own site

BD-1: vision AI powered droid

BD-1: vision AI powered droid

Things used in this project

Hardware components

Software apps and online services

Story

Introduction

LEGO assembly

Raspberry Pi

LED display

Servo control

Azure Custom Vision

Vision AI

Final words

Schematics

How to connect Max7219 and M5Stack servo

Code

Max7219 controller

Servo controller

Vision controller

Github

Credits

Goran Vuksic

Comments

Related channels and tags